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PETITION TO WITHDRAW NOTICE OF ABANDONMENT 



Honorable Commissioner of Patents 

and Trademarks 
Washington, D.C. 20231 

Dear Sir: 

Applicants received a Notice of Abandonment mailed on 
February 25, 2003, copy attached, in the above-identified 
application, "for failure to timely or properly reply to the Notice 
to File Missing Parts (Notice) mailed on 11/08/2001." Applicants 
believe that the Notice of Abandonment was sent in error. 

Applicants filed a response to the Notice on December 28, 
2001, well within the due date of January 8, 2002 for filing a 
response to the Notice. Attached herewith are copies of our 
response, including a copy of our receipt card indicating that a 
Response was filed on December 28, 2001. Also attached is a copy 
of the canceled check for the surcharge required by the Notice. 
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In view of the above, applicants hereby request that the 
Notice of Abandonment be withdrawn. 

It is believed that no fee is due; however, should that be 
incorrect, please charge Deposit Account No. 19-2105 and inform the 
undersigned. 



SHLESINGER, ARKWRIGHT & GARVEY LLP 
3000 South Eads Street 
Arlington, Virginia 22202 
(703) 684-5600 
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NOTICE OF ABANDONMENT UNDER 37 CFR 1.53 (f) OR (g) 



The above-identified application is abandoned for failure to timely or properly reply to the Notice to File Missing 
Parts (Notice) mailed on 11/08/2001. 

• No reply was received. 

A petition to the Commissioner under 37 CFR 1.137 may be filed requesting that the application be revived. 

Under 37 CFR 1.137(a), a petition requesting the application be revived on the grounds of UNAVOIDABLE 
DELAY must be filed promptly after the applicant becomes aware of the abandonment and such petition must be 
accompanied by: (1) an adequate showing of the cause of unavoidable delay; (2) the required reply to the above- 
identified Notice; (3) the petition fee set forth in 37 CFR 1 .1 7(l); and (4) a terminal disclaimer if required by 37 
CFR 1.137(d). 

Under 37 CFR 1.137(b), a petition requesting the application be revived on the grounds of UNINTENTIONAL 
DELAY must be filed promptly after applicant becomes aware of the abandonment and such petition must be 
accompanied by: (1) a statement that the entire delay was unintentional; (2) the required reply to the above- 
identified Notice; (3) the petition fee set forth in 37 CFR 1 . 1 7(m); and (4) a terminal disclaimer if required by 37 
CFR 1.137(d). 

Any questions concerning petitions to revive should be directed to "Office of Petitions" at (703) 305-9282. 
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It is believed that no additional fee is due; however, should 
that be incorrect, please charge Deposit Account No. 19-2105 and 
inform the undersigned. 



SHLESINGER, ARKWRIGHT & GARVEY LLP 
3000 South Eads Street 
Arlington, Virginia 22202 
(703) 684-5600 
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Respectfully submitted, 





Attorney's Docket No.: 7170 



DECLARATION AND POWER OF ATTORNEY - ORIGINAL APPLICATION 



As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated 
below beneath my name: 

I believe I am the original, first and sole inventor (if only one 
name is listed below) or an original, first and joint inventor (if 
plural names are listed below) of the subject matter which is 
claimed and for which Letters Patent is sought on the invention 
entitled 



DEVIC E FOR PICTORIALLY DEPICTING THREE-DIMENSIONAL OBJECTS 



the specification of which is attached hereto 

(check one) X was filed on 27 March 2000 

as Application S.N. PCT/EPOO/02639 
and was amended on 



I hereby state that I have reviewed and understand the contents 
of the above -identified specification, including the claims, as 
amended by an amendment referred to above . 

I acknowledge the duty to disclose information which is material 
to the examination of this application in accordance with Title 
37, Code of Federal Regulations, §1. 56(a). 

I hereby claim foreign priority benefits under Title 35, United 
States Code, §119 (a) -(d) or §365 (b) of any foreign application (s) 
for patent or inventor's certificate, or §365 (a) of any PCT 
International Application which designated at least one country 
other than the United States, listed below and have also 
identified below any foreign application for patent or invention 
certificate, or a PCT International application having a filing 
date before that of the application on which priority is claimed. 

Prior Foreign Application (s) Priority Claimed 



(Number) (Country) (Day/Month/Year Filed) YES NO 



(Number) (Country) (Day/Month/Year Filed) YES NO 
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I hereby claim the benefit under 35 U.S.C. §119 (e) of any United 
States provisional application (s) listed below: 



(Application Ser . No.) (Filing Date) 



(Application Ser. No.) (Filing Date) 

I hereby claim the benefit under Title 35, United States Code, 
§120 of any United States application (s) , or §365 (c) of any PCT 
International application designating the United States, listed 
below and, insofar as the subject matter of each of the claims of 
this application is not disclosed in the prior United States 
application or PCT International application in the manner 
provided by the first paragraph of Title 35, United States Code, 
§112, I acknowledge the duty to disclose material information as 
defined in Title 37, Code of Federal Regulations, §1.56 (a) which 
occurred between the filing date of the prior application and the 
national or PCT international filing date of this application. 



(Application Ser. No.) (Filing Date) (Status) 

(patented, pending, abandoned) 

POWER OF ATTORNEY: As a named inventor, I hereby appoint the 
following attorney (s) and/or agent (s) to prosecute this 
application and transact all business in the Patent and Trademark 
Office connected therewith: 

B. Edward Shlesinger, Jr., Reg. No. 17,225; George A. Arkwright, 
Reg. No. 20,158; George A. Garvey, Reg. No. 17,737; Josef ino P. 
de Leon, Reg. No. 33,166; Terrence L. B. Brown, Reg. No. 32,685; 
Michael M. Zadrozny, Reg. No. 3 0,985; and Brian J. Marton, Reg. 
Agent 3 0,292; 

SEND CORRESPONDENCE TO: SHLESINGER, ARKWRIGHT & GARVEY LLP 

3000 South Eads Street 
Arlington, Virginia 22202 

DIRECT TELEPHONE CALLS TO Josefino P. de Leon (703) 684-5600 

The undersigned hereby authorizes the U.S. attorney (s) or 
agent (s) named herein to accept and follow instructions from 
Hossle & Kudlek as to any action to be taken in the Patent and 
Trademark Office regarding this application without direct 
communication between the U.S. attorney (s) or agent (s) and the 
undersigned. In the event of a change in the persons from whom 
instructions may be taken, the U.S. attorney (s) or agent (s) named 
herein will be so notified by the undersigned. 
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I hereby declare that all statements made herein of my own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that the 
statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or 
imprisonment, or both, under §1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize 
the validity of the application or any patent issuing thereon. 

Full name of sole or first inventor Stef £en SETZER 

Inventor's signature ^ ^2. . &~7 — IMey^ ?.^^ 

Residence Am Wallgraben 142, D-70565 Stuttgart, Germany 

Citizenship German 

Post Office Addr e s s s ame a s above 

Full name of second or joint inventor T^A^o XfEIGEL ' 

Inventor 1 s signature y *//2~i2^ ~Z&v<i \hf / Date V 4 

Residence Am Wallgraben 142, D-70565 Stufafe6art f Germany 
Citizenship German v 

Post Office Address same as above 

Full name of third or j^iiit inventor Dittmar WRIGHT 

Inventor's signature V r J J(Af^ ^ C Jr^/\Jf Date y Z n - (/. o/ 

Residence Am Wallgraben 142, D-70565 Stuttgart, Germany 

Citizenship German 

Post Office Address same as above 
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Date Mailed: 11/08/2001 



NOTIFICATION OF MISSING REQUIREMENTS UNDER 35 U.S.C. 371 IN THE UNITED 
STATES DESIGNATED/ELECTED OFFICE (DO/EO/US) 

The following Kerns have been submitted by the applicant or the IB to the United States Patent and 
Trademark Office as an Elected Office (37 CFR 1.495): 

• U.S. Basic National Fees 

• Indication of SmallEntity Status 

• Copy of IPE Report 

• Copy of references cited in ISR 

• Copy of the International Application 

• Copy of the International Search Report 

• Preliminary Amendments 



The following Items MUST be furnished within the period set forth below in order to complete the 
requirements for acceptance under 35 U.S.C. 371 : 



• Oath or declaration of the inventors, in compliance with 37 CFR 1.497(a) and (b), identifying the 
application by the International application number and international filing date. 

• $65 Surcharge for providing the oath or declaration later than the appropriate 30 months months from 
the priority date (37 CFR 1 .492(e)) is required. 

ALL OF THE ITEMS SET FORTH ABOVE MUST BE SUBMITTED WITHIN TWO (2) MONTH FROM THE 
DATE OF THIS NOTICE OR BY 22 or 32 MONTHS (where 37 CFR 1.495 applies) FROM THE PRIORITY 
DATE FOR THE APPLICATION, WHICHEVER IS LATER. FAILURE TO PROPERLY RESPOND WILL 
RESULT IN ABANDONMENT. 

The time period set above may be extended by filing a petition and fee for extension of time under the 
provisions of 37 CFR 1.136(a). 
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SUMMARY OF FEES DUE: 

Total additional fees required for this application is $65 for a Small Entity: 

• $65 Late oath or declaration Surcharge. 

Applicant is reminded that any communications to the United States Patent and Trademark Office must be 
mailed to the address given in the heading and include the U.S. application no. shown above (37 CFR 1.5) 



ANITA D JOHNSON 



Telephone: (703) 305-3661 
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INFORMATION DISCLOSURE STATEMENT 



Honorable Commissioner of Patents 

and Trademarks 
Washington, D.C. 20231 

Dear Sir: 

Attached herewith is a completed PTO Form- 144 9 and copies of 
the cited references. The Examiner is respectfully requested to 
consider these references and provide a signed copy of the form in 
the next communication in regard to the above- identified 
application. An International Search Report in the corresponding 
PCT application is attached herewith, showing the relevance of the 
references. 
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It is believed that no fee is due; however, should that be 
incorrect, please charge Deposit Account No. 19-2105 and inform the 
undersigned. 



SHLESINGER, ARKWRIGHT & ' GARVEY LLP 
3000 South Eads Street 
Arlington, Virginia 22202 
(703) 684-5600 
lm 



Respectfully submitted, 
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© Method and apparatus for reconstructing three-dimensional objects. 



© One of images of an object (19) captured by at least three cameras (111-1 in) is defined as a basir im»n« 
and a bas.c backprojection line is defined which passes through a feature point in the ^^i™^SS f 
The ba:> n0ted thre r d f imensi - al ^ Point on the object and" the optica, cent of ^ 
^aS C H ma9e ' ? fere r Ce back P ro ' ection «™s « calculated respective feature 

obtamed by projecting the basic backprojection line onto the image by each of the ^J^^^^L 
T^^VT each - a The coor<jinates Qf .^^^ of ^^^^ 
the reference backprojection lines are calculated and the number of the reference ^^^^11^ 
intersecting at each intersection point on the basic backprojection line s c^^ 

nUmbGr ° f interSections is determined to be the position of the noted threSim^ 
Similar processing is performed for all of the feature points in the basic image b^hich the DO iZ " f 
corresponding three-dimensional feature points of the object are determined P S,t '° nS * 
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BACKGROUND OF THE INVENTION 

The present invention relates to measurements of the position, shape and movement of a three- 
dimensional moving object and. more particularly, to a three-dimensional information reconstruction or 
s recovery method and apparatus which can be used in the fields of three-dimensional information re- 
construction, recognition and description (CG) of moving objects. 

Conventional three-dimensional information reconstruction techniques can be classified into three 
categories. A first technique is stereopsis. This technique is one that establishes correspondence of points 
or lines between two right and left images taken by two cameras and estimates, from the positions of the 
to cameras and pairs of corresponding points or lines on the right and left images, the positions of points and 
mes on a scene space corresponding to the pairs of corresponding points or lines. A second technique is a 
three-d.mens.onal information recovery method using a moving camera. This is a method which tracks 
individual feature points on a number of images picked up by a moving camera and estimates the positions 
of points on a scene space corresponding to the feature points. A third technique is a backprojection 
is method. wh.ch recovers or reconstructs a three-dimensional structure of an object in a scene space bv 
projecting back feature points in images to the scene space. 

.With the first technique (see, for example. Kanade T.. Okutomi T. and Nakahara M "A multiple 
baseline stereo method." Proc. Image understanding Workshop, pp. 409-426, 1992 or United States Patent 
No. 4 654.872). many points on the surface of the object tend to be occluded from the sight line of either 
one of the cameras because of uneven object surface, and hence accurate positions of the corresponded 
feature points between the right and left images cannot be obtained, making it hard to obtain highly 
accurate three-dimensional information. The second technique (see. for example, Bolles R C Baker H H 
f I?v/ M f/'r 0nt . D ' H " " E P i P° ,ar -P lane ima 9 e analysis: an approach to determining structure from motion - 
IJCV^ Vo. 1, No. 1, pp. 7-55. 1987) cannot be applied to a moving object, because the object needs to 
stand still during its image capturing session by a moving camera. Recently, there has been proposed a 
technique which permits simultaneous extraction of the three-dimensional shape and motion of an object 
from many images taken by a single camera (see Tomasi C. and Tanade T.: "Shape and motion from 
image streams under orthography: a factorization method," IJCV, Vol. 9, No. 2 pp 137-154 1992) This 
technique recovers three-dimensional information basically by tracking feature points between' the images- 
30 therefore, this method cannot obtain accurate three-dimensional information because the surface of the 
object partly occluded from the sight line of the camera while the camera or the object is moving The 
reason for th.s is that since a noted feature point on the images repeatedly gets out of and into the' view 
.eld, the locus of the feature points on the images frequently breaks, introducing difficulty in tracking the 
feature pomt. Hence, this technique is not suitable for use with a moving object. A third technique is a 
s. houette projection method (see. for example. Ahuja N. and Veenstra J.: "Generation octree from object 
s.lhouettes in orthographic views." IEEE Trans. PAMI, Vol. 11. No. 2. pp. 137-149, 1989). With this method 
however, .t is very difficult to acquire accurate three-dimensional information, because the generation of 
s.lhouette images is extremely difficult and unstable. Another example of the third technique is a method 
which recover or reconstruct edges of a three-dimensional object by extracting edges on images and giving 
a voting to the scene space through use of the extracted edges (see. for example. Hamano T Yasuno T 
and Ishi. K.: "Direct estimation of structure from non-linear motion by voting algorithm without tracking and 
matching. Proc. of ICPR. Vol. 1. pp. 505-508. 1982 and S. Kawato: "3D Shape Recovering by Octree 
To™ 9 S^ n,qUe •" PROCEED,NGS ° f SPLE-The international Society for Optical Engineering 15-16 Nov. 
1992). W.th such a method, however, since a plurality of feature points are simultaneously extracted 
processes for the respective feature points interfere with each other, incurring possibility of a false feature 
point be.ng extracted. A large number of images are needed to solve this problem. For a moving object 
however, much time is consumed to take many images by one camera and a simultaneous image pickup 
system using many cameras is very expensive. 

50 SUMMARY OF THE INVENTION 

It is therefore an object of the present invention to provide a method and apparatus which permit the 
acquisition or recovery of three-dimensional information of an object from its images taken bv a small 
number of cameras. 

A three-dimensional information recovery method according to a first aspect of the invention comprises 

In© Steps I 

(a) wherein an object is imaged by n cameras to obtain n frames of images and these images are output 
as image data, n being an integer equal to or greater than 3; 



as 
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45 
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(b) wherein the image data is received and feature points of the object on the n frames of images are 
respectively extracted; 

(c) wherein one of the n cameras is defined as a basic camera, the image taken by the basic camera as 
a basic image, the other remaining cameras as reference cameras, the images taken by the reference 

5 cameras as reference images, and a backprojection line, which passes through the optical center of the 
basic camera and selected one of feature points in the basic image chosen in correspondence to a noted 
three-dimensional feature point on the object, is defined as a basic backprojection line; 

(d) wherein the basic backprojection line is projected onto the reference images, thereby defining 
epipolar lines on the reference images, respectively; 

10 (e) wherein backprojection lines, which pass through feature points on the epipolar lines on the reference 
images and the optical centers of the reference cameras, are defined as reference backprojection lines; 

(f) wherein the coordinates of intersections of the basic backprojection line and the reference backprojec- 
tion lines are respectively calculated, the numbers of intersections of the basic backprojection line and 
the reference backprojection lines at the respective intersections are counted, and the intersection of the 

is maximum count value is determined to be the position of the above-mentioned noted three-dimensional 
feature point; and 

(g) wherein a sequence of steps (c) through (f) are repeated for each feature point on the basic image to 
obtain the positions of the respective three-dimensional feature points on the object as its three- 
dimensional information. 

20 A method according to a second aspect of the invention comprises the steps: 

(a) wherein an object is imaged by n cameras to obtain n frames of images and these images are output 
as image data, n being an integer equal to or greater than 3; 

(b) wherein the image data is received and feature points on the object on the n frames of images are 
respectively extracted; 

25 (c) wherein one of the n cameras is defined as a basic camera, the image taken by the basic camera as 
a basic image, the other remaining cameras as reference cameras, the images taken by the reference 
cameras as reference images, and a backprojection line, which passes through the optical center of the 
basic camera and a feature point in the basic image chosen in correspondence to a noted three- 
dimensional feature point on the object, is defined as a basic backprojection line; 

30 (d) wherein the basic backprojection line is projected onto the reference images, thereby defining 
epipolar lines on the reference images, respectively; 

.(e) wherein backprojection lines, which pass through feature points on the epipolar lines on the reference 
images and the optical centers of the reference cameras, are defined as reference backprojection lines; 

(f) wherein the coordinates of intersections of the basic backprojection line and the reference backprojec- 
35 tion lines are respectively calculated, the distribution of the intersections along the basic backprojection 

line is filtered by convolution to emphasize the concentration of the distribution of the intersections, and 
the position where the filtered distribution of the intersection is maximum is determined to be the position 
of the above-mentioned noted three-dimensional feature point; and 

(g) wherein a sequence of steps (c) through (f) are repeated for each feature point on the basic image to 
40 obtain the positions of the respective three-dimensional feature points ont he object as its three- 
dimensional information. 

A three-dimensional information recovery apparatus according to the present invention comprises: 

n camera means disposed at different positions, for capturing images of an object and for outputting the 

images as image data, n being an integer equal to or greater than 3; and 
45 image information processing means supplied with the image data from the n camera means, for 

acquiring or recovering three-dimensional information of the object from feature points in the respective 

images; 

the image information processing means comprising: 
feature extracting means for extracting feature points in the images; 
so basic backprojection line generating means wherein one of the n cameras is defined as a basic camera, 
the image taken by the reference camera as a basic image, the other remaining cameras as reference 
cameras and the images taken by the reference cameras as reference images and whereby a backprojec- 
tion line, which passes through the optical center of the basic camera and a feature point in the basic image 
chosen in correspondence to the above-said noted three-dimensional feature point on the object, is 
55 obtained as a basic backprojection line; 

epipolar line generating means for projecting the basic backprojection line onto the reference images to 
obtain epipolar lines; 

reference backprojection line generating means for obtaining, as reference backprojection lines, 
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backprojection lines which pass through feature points on the epipolar lines on the reference imaoes and 
the optical centers of the reference cameras; 

three-dimensional feature extracting means for calculating the coordinates of intersections of the basic 
backprojection line and the reference backprojection lines, for counting the numbers of intersections of the 
s basic backprojection line and the reference backprojection lines at the respective intersections and for 
determining the intersection of the maximum count value to be the position of the noted three-dimensional 
feature point; 

whereby the positions of respective three-dimensional feature points on the object, correspondina to 
respect,ve feature points on the basic image, are obtained as three-dimensional information of the object 
io An apparatus according to a fourth aspect of the present invention comprising: 

n cameras means disposed at different positions, for capturing images of an object and for outputtinq 
the images as image data, n being an integer equal to or greater than 3; and 

image information processing means supplied with the image data from the n camera means for 
acquiring or recovering three-dimensional information of the object from feature points in the respective 
15 imageSt 

the image information processing means comprising: 

feature extracting means for extracting feature points in the images; 

basic backprojection line generating means wherein one of the n cameras is defined as a basic camera 
the .mage taken by the basic camera as a basic image, the other remaining cameras as reference cameras' 
20 the images taken by the reference cameras as reference images and whereby a backprojection line which 
passes through the optical center of the basic camera and a feature point in the basic image chosen in 
correspondence to the above-said noted three-dimensional feature point on the object is obtained as a 
basic backprojection line; 

epipolar line generating means for projecting the basic backprojection line onto the reference imaaes to 
26 obtain epipolar lines; y 

reference backprojection line generating means for obtaining, as reference backprojection lines 
backprojection lines which pass through feature points on the epipolar lines on the reference images and 
the optical centers of the reference cameras; 

filtering means which calculates the coordinates of intersections of the basic backprojection line and the 
reference backprojection lines, respectively, and filters the distribution of the intersections along the basic 
backprojection line by convolution to emphasize the concentration of the distribution of the intersections; 
and * 

three-dimensional feature extracting means whereby the position where the filtered distribution of 
intersection is maximum is determined to be the position of the noted three-dimensional feature point- 
whereby the positions of respective three-dimensional feature points on the object, corresponding to 
respective feature points on the basic image, are obtained as three-dimensional information of the object 

With the three-dimensional moving object information reconstructing methods and apparatuses accord- 
ing to the present invention, since the backprojection area is substantially limited by the introduction of a 
constraint of projective geometry, the interference of backprojection is minimized and three-dimensional 
information can be acquired by backprojection using a small number of images. Accordingly the present 
invention permits reduction of the number of cameras needed to obtain ever-changing three-dimensional 
information of a moving object through simultaneous image capturing by a plurality of cameras. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram illustrating the configuration of an embodiment of the three-dimensional 
information reconstructing apparatus according to the present invention; 

Figure 2 is a processing block diagram for explaining in detail an image information processing part 20 in 

Figure 3 is a backprojection diagram for explaining the principles of the present invention- 
Figure 4 is a three-dimensional graph for explaining the probability of reference backprojection lines 
intersecting at the same point; 

Figure 5A is a perspective view for explaining a pixel quantization error of a feature point in an image; 
Figure 5B is a perspective view for explaining an error of the optical center; 
Figure 6A is a diagram showing the intersection distribution on a basic backprojection line; 
Figure 6B is a graph showing an example of the characteristic of a filter; and 
Figure 6C is a diagram showing an example of a filtered intersection distribution. 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS 

In Fig. 1 there is shown in block form an embodiment of the three-dimensional information reconstruct- 
ing apparatus according to the present invention. The illustrated apparatus is made up of an image input 

5 part 10 and an image information processing part 20. The image input part 10 is composed of n (n being an 
integer equal to or greater than 3) image capturing parts (for example, TV cameras, hereinafter referred to 
simply as cameras) 111 through 11n, A/D converters 121 through 12n, frame memories 131 through 13n, a 
camera controller 14, a movable table 15A, a driver 15B and a movement controller 16. The cameras 111 
through 11n are mounted on the common moving table 15A at different heights. The moving table 15A is 

70 designed to be movable three-dimensionally by the driver 15A formed by a robot arm under the control of 
the movement controller 1 6. 

The cameras 111 through 11n simultaneously capture images of an object 19 at a fixed period (or 
frame period) and each output an analog image signal. When the object 19 is not moving, its images need 
not always be captured simultaneously. The analog signals from the cameras 111 through 11n are 

75 converted by the A/D converters 121 through 12n into digital signals, which are stored, as image data for 
each frame, in the frame memories 131 through 13n. The operations of these cameras, A/D converters and 
frame memories are placed under the control of the camera controller 1 4. The image information 
processing part 20 is supplies with the image data read out of the frame memories 131 through I3n and 
obtains three-dimensional information of the object 19 by performing a process described below with 

20 reference to Fig. 2. 

Figure 2 shows, in the form of functional blocks, the procedure of processing by the image information 
processing part 20 formed by a computer. Position and attitude information of the cameras at the time of 
inputting images of the object 19 by the cameras 111 through 11n is inputted into a camera information 
input part 21 of the image information processing part 20 from the movement controller 16. On the other 

25 hand, image data read out of the frame memories 131 through 13n are fed to a feature extracting part 22. 
The feature extracting part 22 extracts feature points (such as isosbestic points and regions of normal 
images and isosbestic points and regions of thermal images) from the n frames of images inputted from the 
image input part 10. A basic backprojection line generating part 23 selects, as a basic image, an arbitrary 
one of the n frames of images from which the feature points have been extracted and generates a basic 

30 backprojection line which passes through a feature point on the basic image and the optical center of the 
camera having taken the basic image inputted from the camera information input part 21 (which camera will 
hereinafter be referred to as a basic camera). An epipolar line generating part 24 provides epipolar lines 
which are generated by projecting the above-mentioned basic backprojection line onto images (reference 
images) taken by cameras (reference cameras) other than the basic camera. A reference backprojection line 

35 generating part 25 provides all reference backprojection lines which pass through feature points on the 
epipolar lines on the reference images or within a certain distance range on either side of the epipolar lines 
and optical centers of the reference cameras corresponding to the epipolar lines. An intersection extracting 
part 26 calculates the coordinates of all intersections of the basic backprojection line generated by the basic 
backprojection line generating part 23 and the reference backprojection lines generated by the reference 

40 backprojection line generating part 25. An intersection counting part 27 counts the number of intersections 
at intersecting points or regions of the basic backprojection line and the reference backprojection lines 
extracted by the intersection extracting part 26 along the basic backprojection line. A filtering part 28 filters, 
along the basic backprojection line, the intersection count distribution provided by the intersection counting 
part 27. A three-dimensional feature extracting part 29 determines, as the position of a three-dimensional 

45 feature point, one of the intersections which is closest to the peak position where the filtered intersection 
distribution exceeds a fixed threshold value. A three-dimensional movement extracting part 2A calculates 
movement of each three-dimensional feature point and region on the basis of each three-dimensional 
feature point and region information at every time interval extracted by the three-dimensional feature 
extracting part 29. 

so Next, the operation of this invention apparatus of such a construction will be described with reference to 
Figs. 1, 2 and 3. In Fig. 3 inverted images formed on focal planes of the cameras 111 through 11n disposed 
as depicted are imaginarily projected back forwardly of the cameras and shown, for convenience's sake, as 
erect images 32. The camera controller 14 simultaneously activates the n cameras 111 through 11n from an 
input time t, converts the resulting analog image signals by the A/D converters 121 through 12n into digital 

55 image signals and captures them into the n frame memories 131 through 13n to obtain image data of one 
frame in each memory. The camera information input part 21 obtains camera information at the time t 
through calibration (see, for example, J. Weng and P. Cohen and M. Herniou: "Calibration of stereo 
cameras using a non-linear distortion model," Proc. ICCV, pp. 246-253, 1990). 
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By the feature extracting part, feature points on each image 32 (for example, isosbestic points and 
regions in an image, contour lines and regions in a differential image and isothermal points and regions in a 
thermal image) are extracted. In this example, edges of the image are extracted feature points by a Canny 
filter (Canny J. F.:"A computational approachto edge detection," IEEE Trans. PAMI, Vol. 8, No. 6, pp. 679- 
5 698, 1986). Next, the basic backprojection generating part 23 selects an image 32 of an arbitrary one of the 
n frames as a basic image 32s and calculates a backprojection line 34s which passes through one noted 
feature point on the basic image 32s and the optical center 31s of the camera (that is, the basic camera) 
corresponding to the image 32s which is inputted from the camera information input part 21. The 
coordinates (x b , y b , z b ) of an arbitrary point on the backprojection line 34s satisfy the following equation: 



* P - *= y P - y c 2 P - z c ( 1 } 

15 

where (x p y p , z p ) and (x c , y Cl z c ) are coordinates of the noted feature point on the image 32s and the optical 
center 31s of the basic camera, respectively. Such a backprojection line 34s will hereinafter be referred to 
as a basic backprojection line. An arbitrary point E = (x e , y e ) on an epipolar line, which is obtained by 
projecting the basic backprojection line onto each reference image, has such relations as shown below. 

20 

E - R tJ (B / - T ; ) (2) 



25 




T f = (t x , t 2 ) T 

30 

where R r and V are a two-dimensional expression of transposed version of a matrix which defines the 
rotation of the camera optical axis with respect to the coordinate system and a two-dimensional expression 
of a vector which defines the translation of the camera optical axis with respect to the coordinate system, 
respectively, and B' = (x b , y b ) T is a two-dimensional expression of an arbitrary point on the backprojection 

35 line. As is evident from Fig. 3, the basic backprojection line 34s is present in a plane containing the optical 
center 31 of every reference camera and the epipolar line 35 by the reference camera. This plane will 
hereinafter be called an epipolar plane by the reference camera. Three-dimensional feature points on the 
object 19 persent in an extension plane of the epipolar plane of each camera are all projected onto the 
epipolar line 35 of the camera. Next, the reference backprojection line generating part 25 detects all feature 

40 points 33 on the above-mentioned epipolar line 35. One of these feature points 33e on the eipolar line 35 
corresponds to the noted feature point 33s. In practice, however, there are cases where these feature points 
are not on the epipolar line because of an image quatization error. Accordingly, feature point within a fixed 
distance from the epiplar line 35 are also regarded as its feature points 33e. This is followed by obtaining 
backprojection lines 34 which pass through the feature points 33e on the epipolar line 35 in the image 32 

46 by each reference camera and the optical center 31 of the reference camera. These backprojection lines 34 
will hereinafter be referred to as reference backprojection lines. 

Next, the intersection extracting part 26 calculates the coordinates of intersections of the basic 
backprojection line 34s and the reference backprojection lines 34. Owing to the presence of an feature 
extraction error on each image 32 and an error in the optical center of each camera, the situation 

50 occasionally arises where the basic backprojection line 34s and the reference backprojection lines 34 do not 
intersect in the three-dimensional space. In this embodiment, the shortest distance between the basic 
backprojection line 34s and each reference backprojection line 34 is calculated and if this shortest distance 
is smaller than a fixed threshold value, the basic backprojection line 34s and the reference backprojection 
line 34 concerned are regarded as intersecting each other. That is, if the rank of the following matrix with 

55 two arbitrary points B bl = (x b1 , y bt , z bl ) T and B b2 = (x b2 , y b2 , z b2 ) on the basic backprojection line 34s and 
two arbitrary points B rl = (x r1 , y rT , z r1 ) and B r2 = (x r2 , y r2 , z r2 ) on the reference backprojection line 34 is 1, 
these two lines do not extend in parallel to each other. 
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x b2 x bi 

X r2 ~ X rl 



y b2 - y b i 
y r2 - y r i 



Z b2 - 

Z r2 



(3) 



In this instance, two points (x^, ybs, z bs ) T and (x ts , y re , z n ) T , which stay on the basic backprojection line 
34s and the reference backprojection line 34, respectively, and provide the shortest distance between the 
two lines, are given as follows: 
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where: 
K b 
K r 
F b 
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Z r2- Z n 



(4) 



(5) 
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= <G r H - F r G b )/(F b F r - H 2 ) 
= (F b G r - G b H)/(F b F r - H 2 ) 
= <x b2 - x bl ) 2 + (y b2 - y b1 ) 2 + (z b2 - z bl ) 2 
= (x r2 - x r1 ) 2 + (y r2 - y r ,) 2 + (z r2 - z r1 ) 2 

= (x b2 - x bl )(xb1 - x rl ) + (y b2 - y b1 )(y bi - y r1 ) + (z b2 - z b1 )(z b1 - z rl ) 
= (x r2 - x rl )(x rt - x b1 ) + (y r2 - y r i)(y r1 - y b i) + (z r2 - z r1 )(z r1 - z b1 ) 
- (x b2 - x bl )(x rl - x r2 ) + (y b2 - y bl )(y rl - Y r2 ) + (z b2 - z b1 )(z rl - z r2 ) 
If the distance d between these two points satisfies the condition that it be smaller than a threshold value X 
as shown below, 



d = {(x bs - x rs ) 2 + ( Vbs - y rs )2 + ( 2bs - z rs ) 2 } 1/2 < X (6) 

the basic backprojection line 34s and the reference backprojection line 34 on which these two point stay, 
35 respectively, are regarded as intersecting with each other, and the coordinates of the position of the 
intersection 36 are determined to be the midpoint of the distance between the two points. 



x = (x bs + x rs )/2 

y - (yt>s + yr S )/2 

<w z = (z bs + z rs )/2 (7) 

Next, the intersection counting part counts the number of intersections of the basic backprojection line 34s 
and all of the reference backprojection lines 34. 

Incidentally, in the case of using images taken by two cameras (n = 2), there is the possibility of a 

45 plurality of reference backprojection lines 34 intersecting one basic backprojection line 34s at a plurality of 
positions, in such an instance, it cannot uniquely be determined to which intersection the three-dimensional 
feature point corresponds, the latter corresponding to the feature point 33s through which the basic 
backprojection line 34s passes. Now, description will be given of this problem. 

In the interests of clarity, let the number of feature points on the epipolar line of each reference camera 

so obtained by projecting the basic backprojection line 34s onto each reference image be represented by M. 
Only one or none of the M feature points on the epipolar line on the image by each camera correspond to 
the noted feature point. Now, assume that one of the M feature point on each epipolar line corresponds to 
the noted feature point. Accordingly, one reference camera provides M reference backprojection lines and 
the probability that any one of the M intersections of M reference backprojection lines and the basic 

55 backprojection line corresponds to a noted feature point is & = 1/M. The same goes for the other reference 
cameras. Hence, the probability a that one of the reference backprojection lines from every reference 
camera intersects the basic backprojection line 34s at any intersection thereon is a = £ n ~\ When only one 
feature point is present on the epipolar line of the reference camera (that is, when M = 1), that feature point 
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corresponds to the noted feature point and the reference backprojection line intersects the noted feature 
point; in this case, no problem arises. It is when the number M of reference backprojection lines from each 
reference camera is 2 or more that matters. Therefore, discussion will be made below on the case where 0 
= 1/M is in the range of between 0 to 0.5. 
5 Figure 4 is a graph showing the probability a that (n - 1) reference backprojection lines from (n - 1) 

reference cameras intersect at one point on the basic backprojection line when the value 0 is in the range 
of 0 < 0 £ 0.5. As is evident from the graph, even in the case where 0 = 0.5, if the number n of all 
cameras (i.e. the number of viewing points) is equal to or larger than 6, the probability a of five reference 
backprojection lines from reference cameras intersecting at one point on the basic backprojection line is as 
w negligibly small as (0.5) 5 = 0.03. Also when n = 5 (the number of reference cameras is four), the value a 
is as small as 0.06. That is, the possibility that a plurality of reference backprojection lines from respective 
reference cameras happen to intersect at the noted feature point decreases rapidly as the number n of 
cameras increases. Conversely speaking, when the number of reference backprojection lines which actually 
intersect at a certain point on the basic backproojection line is plural (four or more, for instance), it implies 

15 that the intersection is very likely to be a noted feature point Then, it is possible to uniquely determine or 
locate the position of the noted three-dimensional feature point by counting in the intersection counting part 
27 the number of reference backprojection lines intersecting at each point on the basic backprojection line 
34s (hereinafter referred to simply as the intersection number) and then detecting in the three-dimensional 
feature point extracting part 29 the position providing the maximum intersection number, as described 

20 previously. In this way, any given three-dimensional feature point on the object 19 can be made to 
correspond to a unique intersection on the basic backprojection line. 

In view of the above, the value n is set to 3 or larger on the principles of the present invention, but in a 
preferred embodiment six different viewing points, that is, six cameras are used. Such a configuration will 
increase the possibility that a three-dimensional feature point on the object 19, occluded from a particular 

25 one of the viewing points or cameras, can also be observed by or seen from the other cameras and hence 
located. 

In actual measurements, as shown in Figs. 5A and 5B, backprojection lines may sometimes deviate 
from their original directions owing to positional errors of the feature points 33 on the image 32 and of the 
optical center 31 of the camera. For example, since the image 32 composed of image data provided by 
30 each camera has been quantized by pixels of the image, the position of each feature point on the image 
contains a quantization error by the pixels. Now, suppose that the position P of the true feature point 33 on 
the image 32 by a reference camera deviates to P' due to the quantization error as depicted in Fig. 5A. 
Since the reference backprojection line 34 rotates and deviates about the optical center 31. it is determined 
that the true feature point B on the basic backprojection line 34s is at B' (not always on the basic 
35 backprojection line 34s). Alternatively, if the optical center determined by measurement deviates to L' from 
the true optical center L as depicted in Fig. 5B. then the true feature point B on the basic backprojection 
line 34s will be regarded as lying at B". 

Because of the above-mentioned deviation of the reference backprojection line 34 under the influence 
of such kinds of errors, the feature point 33 mapped onto each image from the same three-dimensional 
40 feature point on the object 19 and the reference backprojection line 34 passing through the optical center 
31 corresponding to the feature point 33 are likely to intersect at a point other than the true position of the 
three-dimensional feature point. In such a situation, if the point B' or B" is within a threshold distance from 
the basic backprojection line 34s, it is likely to be counted at an intersection different from the position of 
the true noted feature point on the basic backprojection line 34s at the time of counting the intersection 
45 number by the intersection counting part 27. In this instance, if the position of the intersection which 
provides the maximum count value in excess of a predetermined value is directly extracted, as a three- 
dimensional feature point, by threshold value processing, there is a fear that a three-dimensional feature 
point other than the noted one is extracted as a pseudo or false three-dimensional feature point. 

Since these errors of the backprojection lines scatter randomly in direction and magnitude, however, it 
so can be estimated that the central region of the actually obtainable discrete and expansive count distribution, 
around which intersections concentrate, is most likely to include the true three-dimensional feature point. It 
is therefore possible to decrease the likelihood of extracting pseudo or false three-dimensional feature 
points by filtering the intersection count distribution along the basic backprojection line 34s by the filtering 
part 28 to emphasize the distribution at points where reference backprojection lines concentratedly intersect 
55 the basic backprojection line. To this end, it is possible to use a square filter, a triangular filter, a Gaussian 
filter, a Laplacian-Gaussian filter and so forth. 

For the discrete distribution P(s) of intersection counts 51 along the basic backprojection line 34s, such 
as shown in Fig. 6A, the filtering process using a function f(t) can be done by a convolution based on the 
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following equation. 

P'(s) = /P(s-t).f(t)dt <7) 

5 where P(s) and P*(s) are the distributions of backprojection line intersection counts before and after the 
convolution of the function f(t), respectively, and s is the position along the basic backprojection line 34s. 
The filter function f(t) can be obtained with such various filters as mentioned above. In this embodiment, a 
Laplacian-Gaussian filter (commonly called a V 2 G filter) 52, which has such a characteristic as shown in Fig. 
6B, is used to perform the convolution with the distribution of the intersection counts 51 along the basic 
10 backprojection line 34s. Since the V 2 G filter has a property of emphasizing peaks of a continuous waveform 
while suppressing high-frequency discrete noise, the convolution suppresses discrete intersection counts 
and emphasizes peaks of highly concentrated intersection counts as shown in Fig. 6C. Thus, the extraction 
of false three-dimensional feature point can be prevented by detecting peaks 51 p of the intersection counts 
on the basic backprojection line 34s in the three-dimensional feature extracting part 29 and then determin- 
es ing that the intersection closest to the peak position concerned is the position of the noted three- 
dimensional feature point. 

As described above, intersections of the basic backprojection line 34s and the reference backprojection 
lines 34 from the reference cameras are defined on the basic backprojection line 34s and the number of 
intersections at each intersection is counted. In practice, however, owing to the afore-mentioned optical 

20 center error or pixel quantization error at the feature point, the intersection of one of the reference 
backprojection lines 34 from every reference camera and the basic backprojection line 34s, corresponding 
to the true noted three-dimensional feature point, does not become one in the strict sense. Then, in the 
case of counting the number of intersections at each point of intersection, it is necessary to divide the basic 
backprojection line 34s lengthwise thereof every predetermined small unit length As and count the 

25 intersection number in each divided unit length As. In this case, since the resolution for the intersections 
increases with a decrease in the unit length As, the number of intersections present in the same divided 
unit of length As on the basic backprojection line 34s will decrease. When the unit length As is selected 
small to such an extent that all intersections are separated, the maximum intersection number at each 
intersection goes one and the noted three-dimensional feature point cannot uniquely be identified. Also in 

30 this instance, however, intersections cluster about the noted three-dimensional feature point as shown in 
Fig. 6A, for example; hence, the filtering convolution for the discrete distribution of intersections, such as 
described above, provides a peak at the center of the intersection clustering reagion, making it possible to 
determine that the intersection closest to the peak position is the position of the true three-dimensional 
feature point. 

35 By repeating the above-described process for each of the feature points 33s on the basic image 32s, 
the coordinates of all three-dimensional feature points 36t corresponding to all the feature points 33s on the 
basic image 32s can be determined. In this case, the backprojection lines which intersect at the position 
determined to correspond to the noted three-dimensional feature point are related only to the noted three- 
dimensional feature point. Consequently, by removing the feature points 33 and 33s on the images 32 and 

40 32s corresponding to the three-dimensional feature points determined immediately before obtaining other 
three-dimensional feature points, it is possible to reduce the interference of backprojection and the 
computational cost. Furthermore, by selecting another image as a basic one and repeating the above- 
described process until all images are selected, it is possible to obtain the positions of three-dimensional 
feature points corresponding to feature points on images obtained from all of the viewing points, that is. 

45 taken by all of the cameras used. 

Finally, the three-dimensional movement extracting part 2A extracts three-dimensional movement 
information of the moving object by calculating temporal changes of the three-dimensional feature point 
information available from the three-dimensional feature extracting part 29. A concrete method therefor can 
be implemented by a three-dimensional extended version of a two-dimensional movement information 

so extracting method (see, for example, R. Agarwal and J. Sklansky: "Optical flow estimation by the clustering 
of constraint sets in velocity space," Technical Report TP-91-2, Department of Electrical and Computer 
Engineering, Pattern Recognition and Image Modelling Project, University of Calfornia, Irvine, CA. USA). 

Moreover, it is also possible to carry out tracking of a moving object by repeating operations of: 
inputting the three-dimensional movement information from the three-dimensional movement extracting part 

55 2A into the movement controller 16 of the image input part 10; planning the next viewing point; activating 
the driver 15 to move the plurality of image capturing parts 111 through lln and, at the same time, 
measuring the movement of the driver 15 at an input time t + w; and inputting the measured value into the 
camera information input part. 
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As described above, according to the present invention, since the backprojection area is substantially 
limited by the introduction of a projective geometric constraint, the interference of backprojection is 
minimized, and consequently, three-dimensional information can be obtained by backprojection onto a small 
number of images. Hence, the ever-changing three-dimensional shape of a moving object can be recovered 
by simultaneous image pickup with a small number of cameras. 

It will be apparent that many modifications and variations may be effected without departing from the 
scope of the novel concepts of the present invention. 



Claims 
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1. A method of obtaining three-dimensional information of an object by capturing images of said object by 
a plurality of cameras and making three-dimensional feature points on said object to correspond to 
feature points in said images, said method comprising the steps: 

(a) wherein said object is imaged by n cameras to obtain n frames of images and said images are 
'5 output as image data, said n being an integer equal to or greater than 3; 

(b) wherein said image data is received and feature points of said object on said n frames of images 
are respectively extracted; 

(c) wherein one of said n cameras is defined as a basic camera, the image picked up by said basic 
camera as a basic image the other remaining cameras as reference cameras, the images captured 

20 by said reference cameras as reference images, and a backprojection line, which passes through the 

optical center of said basic camera and a feature point in said basic image chosen in correspon- 
dence to a noted three-dimensional feature point on said object, is defined as a basic backprojection 
line; 

(d) wherein said basic backprojection line is projected onto said reference images to define thereon 
25 epipolar lines, respectively; 

(e) wherein backprojection lines, which pass through said feature points on said epipolar lines on 
said reference images and optical centers of said reference cameras, are defined as reference 
backprojection lines; 

(f) wherein the coordinates of intersection points of said basic backprojection line and said reference 
30 backprojection lines are calculated, the numbers of intersections of said basic backprojection line 

and said reference backprojection lines at their respective intersection points are counted, and the 
intersection point of the maximum count value is determined to be the position of said noted three- 
dimensional feature point; and 

(g) wherein said steps (c) through (f) are repeated for each of said feature points on said basic 
35 image to obtain the position of said three-dimensional feature points on said object as its three- 
dimensional information. 

2. A method of obtaining three-dimensional information of an object by capturing images of said object by 
a plurality of cameras and making three-dimensional feature points on said object to correspond to 
40 feature point in said images, said method comprising the step: 

(a) wherein said object is imaged by n cameras to obtain n frames of images and said images are 
output as image data, said n being an integer equal to or greater than 3; 

(b) wherein said image data is received and feature points of said object on said n frame of images 
are respectively extracted; 

45 (c) wherein one of said n cameras is defined as a basic camera, the image taken by said basic 

camera as a basic image, the other remaining cameras as reference cameras, the images captured 
by said reference cameras as reference images, and a backprojection line, which passes through the 
optical center of said basic camera and a feature point in said basic image chosen in correspon- 
dence to a noted three-dimensional feature point on said object, is defined as a basic backprojection 

so line; 

(d) wherein said basic backprojection line is projected onto said reference images to define thereon 
epipolar lines, respectively; 

(e) wherein backprojection lines, which pass through said feature points on said epipolar lines on 
said reference images and optical centers of said reference cameras, are defined as reference 

55 backprojection lines; 

(f) wherein the coordinates of intersection points of said basic backprojection line and said reference 
backprojection lines are calculated, the distribution of said intersection points along said basic 
backprojection line is filtered by convolution to emphasize the concentration of said distribution of 
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said intersection points, and the position where said filtered distribution of the intersection points is 
maximum is determined to be the position of said noted three-dimensional feature point; and 
(g) wherein said steps (c) through (f) are repeated for each of said feature points on said basic 
image to obtain the position of said three-dimensional feature points on said object as its three- 
5 dimensional information. 

3. The method of claim 1 or 2, wherein a position said basic backprojection line is defined every 
predetermined unit length of said basic backprojection line lengthwise thereof and the number of 
intersections at each of said intersection point is the number of intersections in said unit length at the 

io position on said basic backprojection line through which said reference backprojection lines pass. 

4. The method of claim 3, wherein said reference backprojection lines intersecting with said basic 
backprojection line are reference backprojection lines lying within a predetermined range of distance 
from said basic backprojection line. 

15 

5. The method of claim 1, wherein said step of counting said number of intersections is a step wherein 
the distribution of said intersections along said basic backprojection line is filtered by convolution to 
emphasize the concentration of said distribution of said intersections and the position where said 
filtered distribution of said intersections is maximum is determined to be the position of said noted 

20 three-dimensional feature point. 

6. The method of claim 2 or 5, wherein said filtering is performed by a Laplacian-Gaussian filter. 

7. The method of claim 2 or 5, wherein said filtering is performed by a square filter. 

25 

8. The method of claim 2 or 5, wherein said filtering is performed by a triangular filter. 

9. The method of claim 2 or 5 f wherein said filtering is performed by a Gaussian filter. 

30 10. The method of claim 1 or 2, wherein said step (g) includes a step wherein each time said feature point 
corresponding to said noted three-dimensional feature point is identified by performing said series of 
steps (c) through (f), said identified feature point is removed from candidates for processing in the 
subsequent repetition of said steps. 

35 11. The method of claim 1 or 2, wherein the number n of said cameras is in the range of from 5 to 8. 

12. The method of claim 1 or 2, further comprising a step wherein upon each completion of said step (g), 
said step (g) is newly repeated for every one of said cameras, using one of said reference cameras as 
a new basic camera and said basic camera as a new reference camera. 

40 

13. The method of claim 1 or 2, further comprising a step wherein upon each repetition of simultaneous 
capturing of images of said object by said n cameras at a fixed period, three-dimensional information of 
said object is obtained following said series of steps. 

45 14. A three-dimensional information reconstructing apparatus comprising: 

n camera means disposed at different positions, for capturing images of an object and for 
outputting said images as image data, said n being an integer equal to or greater than 3; and 

image information processing means supplied with said image data from said n camera means, for 
acquiring three-dimensional information of said object from feature points in said images; 
so said image information processing means comprising: 

feature extracting means for extracting said feature points in said images; 

basic backprojection line generating means wherein one of said n camera means is defined as a 
basic camera, the image captured by said basic camera means as a basic image, the other remaining 
camera means as reference camera means, the images captured by said reference camera means as 
55 reference images, and whereby a backprojection line, which passes through the optical center of said 
basic camera means and a feature point in said basic image chosen in correspondence to a noted 
three-dimensional feature point on said object, is obtained as a basic backprojection line; 

epipolar line generating means for projecting said basic backprojection line onto said reference 
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images to obtain epipolar lines; 

reference backprojection line generating means for obtaining, as reference backprojection lines, 
backprojection lines which pass through feature points on said epipolar lines on said reference images 
and the optical centers of said reference camera means; and 

three-dimensional feature extracting means for calculating the coordinates of intersection points of 
said basic backprojection line and said reference backprojection lines, for counting the number of 
intersections of said basic backprojection line and said reference backprojection lines at each of said 
intersection points and for determining that one of said intersection points which has the maximum 
count value is the position of said noted three-dimensional feature point; 

whereby the positions of respective three-dimensional feature points on said object, corresponding 
to respective feature points on said basic image, are obtained as said three-dimensional information of 
said object. 

15. A three-dimensional information reconstructing apparatus comprising: 

n camera means disposed at difference positions, for capturing images of an object and for 
outputting said images as image data, said n being an integer equal to or greater than 3; and 

image information processing means supplied with said image data from said n camera means, for 
acquiring three-dimensional information of said object from feature points in said images; 
said image information processing means comprising: 
feature extracting means for extracting feature points in said images; 

basic backprojection line generating means wherein one of said n camera means is defined as a 
basic camera means, the image captured by said basic camera means as a basic image, the other 
remaining camera means as reference cameras, the images captured by said reference camera means 
as reference imaged, and whereby a backprojection line, which passes through the optical center of 
said basic camera means and a feature point in said basic image chosen in correspondence to a noted 
three-dimensional feature point on said object, is obtained as a basic backprojection line; 

epipolar line generating means for projecting said basic backprojection line onto said reference 
images to obtain epipolar lines; 

reference backprojection line generating means for obtaining, as reference backprojection lines, 
backprojection lines which pass through feature points on said epipolar lines on said reference images 
and the optical centers of said reference camera means; 

filtering means which calculates the coordinates of intersection points of said basic backprojection 
line and said reference backprojection lines, respectively, and filters the distribution of said intersection 
points along said basic backprojection line by convolution to emphasize the concentration of said 
distribution of said intersection points; and 

three-dimensional feature extracting means whereby the position where said filtered distribution of 
said intersection points is maximum is determined to the position of said noted three-dimensional 
feature point; 

whereby the positions of respective three-dimensional feature points on said object, corresponding 
to respective feature points on said basic image, are obtained as said three-dimensional information of 
said object. 

16. The apparatus of claim 14, wherein said feature extracting means includes filter means for emphasizing 
the concentration of said distribution of intersections by convolution along said basic backprojection line 
and the position where said filtered distribution of intersections is maximum is determined to be the 
position of said noted three-dimensional feature point. 

17. The method of claim 15 or 16, wherein said filter means is a Laplacian-Gaussian filter. 

18. The apparatus of claim 15 or 16, wherein said filter means is square filter. 

19. The apparatus of claim 15 or 16, wherein said filter means is a triangular filter. 

20. The apparatus of claim 15 or 16, wherein said filter means is a Gaussian filter. 

21. The method of claim 14 or 15, wherein said number n of cameras means is in the range of from 5 to 8. 
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22. The apparatus of claim 14 or 15, further comprising camera control means for repeating simultaneous 
image capturing of said object by said n camera means at a fixed period and for supplying said image 
data to said image information processing means upon each generation of said image data. 
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The invention provides a three-dimensional digital scanner which includes a multiple view detector which is responsive to a broad 
spectrum of visible light. The multiple view detector is operative to develop a plurality of images of a three-dimensional object which 
is being scanned. The plurality of images are taken from a plurality of relative angles with respect to the object, and the plurality of 
images depict a plurality of surface portions of the object. A digital processor including a computational unit is coupled to the detector 
and is responsive to the plurality of images so that it develops 3-D coordinate positions and related image information for the plurality of 
surface portions of the object. A three-dimensional image of the object to be scanned is thus developed by the digital processor. The data 
developed includes both shape and surface image color information. 
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Technical Fi>.M 

5 This invention relates generaJ.y to optica, scanners, and more particularly to optical 

scanners for providing a digital representation of three dimensional objects. 

Background Art 

Methods for successfully obtaining «wo dimensional ("2-D") color image data for objects 
10 have been developed. This proC ess is commonly known as ■„„ dimensional scanning or 
digmztng. When an object is scanned, a digitai data file is created which contains image data 
■ncludmg color informatton which is associated with a se, of two dimensional points or 
coordmates. The color information is obtained by an optica, detector or set of optical detectors 
that are typically organized in a one or two dimensional array. 
1 5 Matching the color information with me correct two dimensional point or location is no. a 

significant problem in two dimensional scanning since the two dimensional point on which the 
optical detector is focused is the same point mat is asscciated with the color information obtained 
by the detector. The color information is mis.ocated only to the extent that mere is some error in 
the location of the point on which the detector is focused (e.g. an error introduced by the optical 
2 0 system) and that error can readily be minimized. 

The problem of associating color information with three dimensional ("3-D-) objects is 
not so easily solved. This is because prior art methods obtain color information with a two 
dimensional scanning method, while position information is obtained by a three dimensional 
scanning method. The mapping of the 2-D color information to the 3-D position information is a 
2 5 complicated process which is prone to significant error. 

Many methods exist for obtaining the three dimensional location of the surface points of 
the object. One such method is a system which uses a laser range finder ,„ scan the object and 
record the distance between the known three dimensional location of the range finder and the 
measured location of the surface of the object. The tesul, of using this method or other methods 
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of generating three dimensional surface models is a set of three dimensional points which 
accurately represent the surface of the object. A characteristic of this method and other methods 
of obtaining a three dimensional surface model is that it is inherently monochromatic, that is, no 
color information is obtained in the process. If three dimensional color information is desired, 
5 then it must be generated by somehow combining or conformally mapping the two dimensional 
color information onto the three dimensional surface model. 

The problem of conformally mapping the two dimensional color information onto the 
three dimensional surface model is difficult and it is common for mismatching of color 
information with the three dimensional points to occur. The problem may be visualized by 

1 0 imagining a white statue or bust of a person's head and a color photograph of the same person's 
face. The photograph cannot simply be projected onto the bust to transfer the correct color 
information to the correct points on the bust or significant distortion will occur. A significant 
amount of judgment must be exercised in order to correctly associate the color information from 
the photograph with the correct surface points on the bust. Similarly, it is difficult to accurately 

1 5 associate color information obtained from two dimensional optical detectors with the correct 
points on a three dimensional surface model. Another problem in the prior art is that color 
information is not used to determine surface locations, which means less than the total amount of 
information that is available is being used. Furthermore, both a 2-D and 3-D system is required, 
which adds cost. 

20 What is needed is a way of generating a set of three dimensional points representing a 

surface in such way that the three dimensional points are already associated with color data so that 
conformally mapping separately generated color data onto the set of three dimensional surface 
points is not necessary. Furthermore, it is desirable to utilize all available frequencies of light to 
determine surface point positions to maximize the accuracy of the scanning process and to 

25 eliminate a separate 3-D scanning step. 
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Disclosure of the Invention 



Accordingly, the present invention provides a system and method for using the color 
information from a series of two dimensional color images to derive the three dimensional 
5 location in space of the surface points which produced the color images. Because the color 
information itself is used to derive the three dimensional location of the surface points, there is no 
need to conformally map separately generated color information onto the derived three 
dimensional surface points. The points are derived from color information and so are already 
associated with the correct color information. Also, the use of the color information increases the 
1 0 accuracy of the three dimensional location of the surface points. 

In one embodiment, the present invention provides a three dimensional digital scanner 
which includes a multiple view detector which is responsive to a broad spectrum of visible light. 
The multiple view detector is operative to develop a plurality of images of a three dimensional 
object which is being scanned. The plurality of images are taken from a plurality of relative 
angles with respect to the object, and the plurality of images depict a plurality of surface portions 
of the object . A digital processor including a computational unit is coupled to the detector and is 
responsive to the plurality of images so that it develops 3-D coordinate positions and related 
^ image information for the plurality of surface portions of the object. A three dimensional image 
I of the object to be scanned is thus developed by the digital processor. The data developed 
20 I; includes both shape and surface image color information. 

In another embodiment, a three dimensional color digital scanner includes a color detector 
responsive to a broad spectrum of visible light to develop a plurality of images of a three 
dimensional object . A rotary object support having an axis of rotation allows the detector to 
develop a plurality of images of a three dimensional object. The plurality of images depict a 
plurality of surface portions of the object. A digital computer is coupled to the detector. The 
computer tracks patches of the surface portions of the object to determine coordinates of the 
patches as a function of the rotation of the rotary object support and determines radii of the 
patches from the axis of rotation. 
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In another embodiment, a method for scanning a three .dimensional object includes 
developing a plurality of images of a three dimensional object taken from a plurality of relative 
angles with respect to the object. The plurality of images depict a plurality of surface portions of 
the object to be scanned. 3-D coordinate positions and related image information about the 
5 plurality of surface portions of the object is computed from the plurality of images such that a 

| three dimensional image of the object is developed that includes both shape and surface image 

i 

* information. 

In another embodiment, a method for determining three dimensional coordinates of a 
surface portion of an object includes obtaining a plurality of images of the surface portion of the 

1 0 object and identifying a trackable patch of the surface portion in an initial image. An initial set of 
two dimensional coordinates of the trackable patch in the initial image is determined along with at 
least one additional set of two dimensional coordinates of the trackable patch in another of the 
images. A radial coordinate of the trackable patch is determined and then a set of three 
dimensional coordinates of the trackable patch are determined from the radial coordinate of the 

1 5 trackable patch. 

In another embodiment, a method for determining three dimensional coordinates of a 
surface portion of an object includes rotating the object about an axis of rotation so that a plurality 
of images of the surface portion of the object are obtained as the object is rotates about the axis of 
rotation. A trackable patch is identified and the two dimensional coordinates of the trackable 

20 patch are determined. The movement of the trackable patch is tracked as a function of the 
rotation of the object. A radial distance of the trackable patch from the axis of rotation is 
determined based on the movement of the trackable patch as a function of the rotation of the 
object and three dimensional coordinates of the surface portion of the object are derived from the 
coordinates of the trackable patch and the radial distance of the trackable patch from the axis of 

25 rotation. 

The present inventions provides a system and method for obtaining 3-D surface 
information that is linked to color information without the need to conformally map 2-D color data 
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onto a 3-D surface. The accuracy of the system is enhanced by the use of color data and the cost 
of the system is reduced because the 3-D surface is derived from a series of 2-D images. These 
and other advantages of the present invention will become apparent upon reading the following 
detailed descriptions and studying the various figures of the drawings. 
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Brief Description of the Drawings 



Figure I illustrates a system for obtaining a series of two dimensional color images of an 
object and processing those images to obtain a three dimensional model of the surface of the 
5 object. 

Figure 1 A illustrates an alternative embodiment of the present invention which enables the 
top and bottom portions of an object to be scanned. 

Figure IB illustrates another embodiment of the present invention which produces 
enhanced shading of an object. 
10 Figure 1C illustrates an arrangement where a detector is translated about a stationary 

object. 

Figure ID illustrates an embodiment of the present invention which uses a multiple 
number of detectors instead of moving a single detector. 

Figure 2 illustrates in detail an architecture of an image acquisition system. 
1 5 Figure 3A is a flow diagram illustrating a process of obtaining multiple images of a 

rotating object. 

Figure 3B is a flow diagram illustrating a process for generating three dimensional surface 
data from the two dimensional images of the object. 

Figure 4 is a flow diagram illustrating a process performed on the images before they are 

20 stored. 

Figure 5A illustrates the vector nature of the color data obtained. 

Figure 5B illustrates an example of blue color data at times 0, 1,2, and 3 for a line of 

pixels. 

Figure 5C illustrates how the data can be compressed by recording only the changes in the 
25 color data. 

Figure 6 is a flow diagram illustrating a process for identifying the silhouette of the object 
in each image. 

Figure 7 is a flow diagram illustrating a process for finding silhouette edges along each 
scan line. 
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Figure 8 is a flow diagram illustrating a process for determining e a set of trackable 
patches. 

Figure 8A illustrates how to search an image for trackable patches. 

Figure 9 is a flow diagram illustrating a process for determining the radius of the location 
5 of patches on the surface of the object as the object is rotated. 
Figure 9A illustrates a set of patch tracking limits. 

Figure 9B illustrates the motion of trackable patches in different images with different 
angular displacements. 

Figure 9C illustrates the determination of an exact position of the patch in an image. 
1 0 Figure 9D is a graph which illustrates the filtering of raw data points. 

Figure 9E is a graph which illustrates how the radius is determined from the points 
representing the path of the trackable patch across angularly displaced images. 

Figure 10 is a flow diagram illustrating t he post processing that occurs once the radius of 

the trackable patch is known. 
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Best Modes for Carrying out the Invention 
In Figure 1 , an embodiment of the present invention includes a system for obtaining a 
series of two dimensional color images of an object and processing those images to obtain a three 
dimensional model of the surface of the object. An object 100 which is to be digitized is placed 
5 on a rotatable platform 102. A motor 104 is provided to drive rotatable platform 102 via a shaft 
106. A position encoder 108 detects the angular position of rotatable platform 102 and generates 
an electrical signal which represents the angular position of rotatable platform 102. An optical 
detector 1 10 (e.g. a color video camera) views object 100 and creates a two dimensional color 
image of object 100. 

10 As object 100 is rotated by rotatable platform 102, detector 1 10 captures a series of color 

images of object 100. Each color image taken at a different time is associated with an angular 
rotation of object 100 about an axis of rotation, "A" which runs through shaft 106. Information 
about the angular position of object 100 is obtained from position encoder 108. Thus, each 
"snapshot" or image of object 100 taken by detector 1 10 from a different view is associated with 

15 data about the angle of rotation of object 100 with respect to detector 110. An image input 
processing system 120 ("computer") controls the image acquisition process and records the 
acquired images along with the associated angular position data. That is, processing system 120 
is connected to detector 1 10 and receives data for each image or snapshot taken of object 100 
from detector 110, and position encoder 108 sends angular position information to processing 

20 system 120, so that processing system 120 can associate the image data from detector 110 with 
the angular position data taken at the same time. In other embodiments, detector 1 10 is a film 
camera and processing system 120 receives data from a digitizer which digitizes the film images r 
from detector 1 10. 

Processing system 120 includes a processing unit 122 and a monitor 124 and also 
25 controls motor 104. A monitor 124 can display a current image 126 being captured by detector 
1 10 or other information about the capturing process. 

Once processing system 120 has obtained a series of images, those images are transferred 
to an image processor 130 ("computer"). Image processor 130 can receive data from processing 
system 120 in a number of different ways. Image processor 130 can be directly connected to 
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processing system 120 via direct connection 132, or data from processing system 120 can be 
transferred to a removable storage medium such as disk 134 which may be read by image 
processor 130. Processing system 120 may also transfer data to image processor 130 via the 
Internet or a modem connection. Image processor 130 includes processing unit 136 and also 
5 includes monitor 138. 

In other embodiments, processing system 120 and image processor 130 are combined on 
a single computer. The advantage of separating the functions of processing system 120 and 
image processor 1 30 is that the data acquisition and storage function performed by processing 
system 120 and control of the data acquisition system does not require a complex or powerful 
10 processor. On the other hand, image processor 130 receives data representing a series of two 
dimensional images and perform complex and computationally intensive operations on that data to 
produce a three dimensional surface model. Image processor 130 is therefore, given current 
technology, likely to be a more powerful (and costly ) computer than processing system 120. If 
that is the case, then it is economically beneficial to utilize a large number of relatively cheap 
1 5 processors for data acquisition and temporary storage and send data from those relatively cheap 
systems to a smaller number of image processors which generate the three dimensional surface 
model from the set of two dimensional color images. 

Figure 1 A illustrates an alternative embodiment of the present invention which enables the 
top and bottom portions of an object to be scanned. Again, object 100 is supported by rotatable 
20 platform 102 which is driven by motor 104. In this embodiment, shaft 107 engages the edge of 
rotatable platform 102, so that motor 104 and shaft 107 do not obscure the image of the bottom of 
object 100. Rotatable platform 102 is made from a transparent material so that the bottom of 
object 100 may be viewed through rotatable platform 102. A set of mirrors 109 are placed within 
the field of view of detector 1 10 so that images of the top and bottom surfaces of object 100 are 
25 captured by detector 1 10 in addition to the side views. 

Figure IB illustrates another embodiment of the present invention which is designed to 
produce contrast enhancing shading of object 100. Again, object 100 is supported by rotatable 
platform 102 which is driven by a motor 104 via a shaft 106. A second motor 142 also drives a 
rotatable platform 144 via shaft 146. Encoder 148 generates data representative of the rotational 
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position of rotatable platform 144 and transmits that data to processing system 120. Likewise, 
motor 142 receives control commands from processing system 120. A light 150 is mounted on 
rotatable platform 144 to provide illumination of object 100. Light 150 is oriented to provide 
contrasting illuminated and shaded portions on object 100 which aid in the tracking of features on 
5 the surface of object 100. Because light 150 is mounted on rotatable platform 144 which is 
separately controllable by processing system 120, different orientations of light 150 with respect 
to object 100 may be checked to determine which one best enhances the surface features of object 
100. When platforms 102 and 144 are rotated in a synchronized manner, the shading remains 
constant. Additionally, multiple sets of views of object 1 00 with different shadings can also be 
1 0 obtained by changing the relative position of platforms 102 and 144. 

Figures 1, 1A, and IB each depict embodiments where in the object being imaged is 
rotated. In another embodiment of the present invention, the object remains stationary and the 
detector moves around the object. Figure 1C illustrates an arrangement where a detector is 
translated about a stationary object. It should be noted that as the detector 1 10 is moved, the 
1 5 optics 1 1 1 remain pointed at the object 100. Detector 1 10 can be move in many ways and object 
100 can be supported in many ways. In one embodiment, an unobstructed view of object 100 is 
obtained by suspending it from very thin wires. Detector 110 is translated about object 100. If 
object 100 is very large, detector 1 10 could be mounted on, for example, a helicopter and flown 
around object 100. It is not necessary that the motion of detector 1 10 be exactly circular around 
20 object 100. The angular and radial components of the motion of detector 1.10 with respect to 
object 100 can be computationally analyzed, as will be appreciated by those skilled in the art. As 
long as the position of detector 110 is measured and recorded, the relative angular position of 
detector 1 10 with respect to object 100 can be determined for each image taken by detector 1 10. 
Methods of determining the position of detector 1 10 include using GPS or a laser positioning 
25 system. Once the angular component of the motion is analyzed and the radial component is 
calculated, the system compensates for the radial component and the images generated by detector 
1 10 can be processed similarly to the images generated by a system that includes a rotating object 
and a stationary detector. 
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Figure ID illustrates an embodiment of the present invention which uses a multiple 
number of detectors instead of moving a single detector. A top view of object 100 is shown and 
set of detectors 1 10 are provided at different angular displacements with respect to object 100. 
The advantage of this embodiment is that no motion is required and the need for motors, 
5 encoders, and rotatable supports is limited. The image of object 100 captured by each detector is 
angularly displaced with respect the images captured by the other detectors and so the images may 
be processed in a similar manner as successive images taken by one moving detector. The 
cost of multiple detectors 1 10 may be less than the cost of a rotatable drive or a mechanism for 
moving detector 110 and recording the position of detector 1 10. Another advantage of this 
1 0 approach is that all of the images of object 100 can be created simultaneously. 

Figures 1 through ID depict various embodiments for creating multiple images of object 
100 with object 100 and detector 1 10 at different relative angular displacements. Each of these 
systems provide two dimensional color images of object 100 observed at different angles. This 
two dimensional information is converted into a three dimensional surface model of object 100 by 
1 5 the process and apparatus of the present invention. 

Figure 2 illustrates in detail the architecture of processing system 120 used in some 
embodiments. A microprocessor 200 is connected to a memory bus 202 and memory bus 202 is 
connected to a RAM 204 and a ROM 206. Microprocessor 200 is also connected to an 
input/output ("I/O") bus 208. A video interface 210 is coupled to I/O bus 208 to control monitor 
20 124, as is detector interface 212. Detector interface 212 buffers and processes data from the 
detector and also carries output commands to the detector from microprocessor 200. In certain 
embodiments where a moving detector is used, the detector provides its own control and records 
its own position. In such embodiments, the detector/processor interface need only be capable of 
transferring data from the detector, including both image and detector position data, to the 
2 5 processor storage system. 

Mass storage 214 (such as a hard disk drive) is also connected to input/output bus 208 
and provides storage capacity for the multiple images generated by the optical system. 
Removable storage 216 (such as a floppy disk drive) also provides a way of transferring data 
files to and from processing system 120 and another processing system. Alternatively, 
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communications interface 218 can be used to transfer files as well. Communications interface 
218 may be connected to a local area network ("LAN") or wide area network ("WAN") for 
communication with other workstations. Position controller 220 is connected to input/output bus 
208 and provides control to a motor in embodiments where processing system 120 provides 
5 control commands for rotating object 100. In such embodiments, position detector 222 receives 
data from an encoder so that processing system 120 may keep track of the position of object 100. 
Lighting control 224 is also connected to input/output bus 208 and is used to control the position 
of lights which may be moved with respect to object 100. Lighting control 224 also controls the 
intensity of those lights. 

10 The architecture shown for processing system 120 in Figure 2 is capable of supporting 

any of the embodiments shown in Figures 1 -ID. If the object is to be rotated, position controller 
220 and position detector 222 provide control of the rotation. Position information about object 
100 can be integrated with image data from interface 212 and stored in mass storage 214. 
Movement and intensity control of the light is controlled by lighting control 224. If an 

1 5 autonomous detector is used, data about the detector position and images captured by the detector 
can be transferred to processing system 120 via communications interface 218 or removable 
storage 216. Multiple detector interfaces are provided to control a multiple number of detectors in 
embodiments which use more than one detector. As described above, a three dimensional surface 
model can be computed using microprocessor 200 and the data contained in mass storage 214, 

20 or, alternatively, the data in mass storage 214 can be transferred to a more powerful image 
processing system. 

Figure 3A is a flow diagram for the process of the present invention of obtaining multiple 
images of a rotating object. Preferably, the method is implemented on a processing system 120. 
The process starts at step 300, and the user places the object on the rotatable platform in step 302. 
25 The object begins to rotate while it is being imaged by a detector. In step 304, the processor 
checks whether the required of number images have already been captured or taken. If the 
required number of images have been captured, then the process is finished at step 306. The two 
dimensional image data is then ready to be taken to an image processor for generation of a three 
dimensional surface model. If more images are to be captured, then control is transferred to step 
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308, and a command is sent to the detector to capture an image. The image is preferably 
preprocessed and stored in step 310, and control is transferred back to step 304. Once the 
process depicted in Figure 3 A is complete, a set of two dimensional color images of the object 
has been obtained at different relative angular displacements between the detector and the object. 
5 Figure 3B illustrates the process implemented on image processor 130 for generating 

three dimensional surface data from the two dimensional images of the object developed by the 
process illustrated in Figure 3 A which was implemented on processing system 120. The process 
begins at step 350 and in a step 352, the processor identifies the silhouette of the object in each of 
the two dimensional color images of the object provided by the image acquisition system. This 
1 0 silhouette information will be used to reduce the area in each image that must be processed. In 
step 354, a set of tracking points is determined within the silhouettes of the object. These 
tracking points are associated with surface features of the object which are identifiable in different 
images of the object taken by the optical detector. The portion of the image of the object which 
corresponds to the identifiable feature is referred to as a trackable patch. Each trackable patch 
1 5 consists of a group of pixels surrounding a geometric point on the surface which is being 
scanned. The trackable patch is thus a feature associated with and proximal to the geometric 
point. The patch is distinguishable from adjacent surface locations and therefore trackable. The 
trackable patch is tracked through a series of images of the object taken at different relative 
angular displacements between the object and the detector. Step 356 develops a radius from the 
20 axis of rotation for each trackable patch which will account for the motion of the trackable patch 
in the images. The radius of the trackable patch is used to approximate the radius to the geometric 
point associated with the trackable patch. 

Once step 356 is completed, a three dimensional surface consisting of the points which 
correspond to each trackable patch and interpolations between those points has been generated. 
25 In step 358, coordinates of the points corresponding to the modeled surface are output in a 
standard format. In some embodiments, the raw trackable patch points are output. In other 
embodiments, points are interpolated between the raw trackable patch points, so that a regular 
array of points representing a three dimensional surface is generated. Each trackable patch point 
is already associated with a color value because the trackable patch points are associated with the 
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trackable patch color information. The color value of the trackable. patch is simply attributed to 
the point determined for that patch. Step 358 outputs the coordinates and their color values. The 
process is completed at 360. 

The process of Figure 3B generates three-dimensional surface data from a set of a two 
5 dimensional images in a manner that associates color information with the three dimensional 
points during the generation of those points. Indeed, the color information is used to identify the 
points which are tracked thereby enhancing the tracking process. The system does not identify 
the surface of the object independent of color image information and then match the surface points 
to color information. In the system depicted in Figure 3B, optical detectors obtain color images 
1 0 of the object. The 3-D surface points of the object are not determined independently from that 
color image information. In fact, the surface points of the object are not directly determined, 
rather, the locations of the features located on the surface are determined. Those surface features 
include color information. Therefore, the locations of color features are directly determined and 
do not need to be mapped onto a surface model which is independently generated. The regularly 
15 spaced points of the surface model are then generated from the points representing the color 
surface features by interpolating between the surface feature points. 

Figure 4 illustrates a process 310 that can be performed on the images before they are 
stored. The process starts at 400. Step 402 analyzes the difference between the current acquired 
image and the previous acquired image. Step 404 applies a compression technique to the data to 
20 reduce the amount of storage space taken up by the data. The process is finished in step 406. 
Thus, in this embodiment using process 3 1 0, the data is compressed for more compact digital 
storage. 

Figures 5A, 5B, and 5C further illustrate the color image pixel data and a compression 
technique implemented in one embodiment of the invention. Figure 5A illustrates the vector 
25 nature of the color data obtained. In the example, shown, red-green-blue ("RGB") color data is 
used. This or any other color scheme such as cyan-magenta-yellow black ("CMYK") can be 
implemented in accordance with the present invention. The three axes shown in Figure 5A are 
labeled R, G, and B. Color values can be represented as vectors directed to points in this color 
space. For example, vector 500 represents the color value of a pixel. The values of R, G, and B 

14 

BNSDOCID: <WO 9817970A1J_> 



WO 98/17970 




PCT/US97/19331 



are represented by the point to which vector 500 is directed. Vector. 500 is the sum of the green 
component vector 502, the blue component vector 504, and the red component vector 506. 
Thus, for each point, there is a vector in color space describing the color of the point. The 
position or pixel location of the point is stored along with the three dimensions of color 
5 information which represent the intensities of the three color signals, RGB. 

Figure 5B illustrates an example of what the color data for the color blue looks like at 
times 0, 1,2, and 3 for a line of pixels. At time 0, no blue color surface feature is being imaged 
by the line of pixels. At time 1, a surface feature with a blue component has moved into the 
position being imaged by the line of pixels, so that 8 blue pixels are observed after X number of 
10 no blue pixels are observed. Following the 8 blue pixels, Y pixels of no color are again 
observed. In this example, only one intensity of blue is shown for the purpose of simplicity. In 
an actual example, varying intensities of blue could also be observed and recorded. At time 2, the 
surface feature has moved to the right and now X + 2 pixels are observed with no blue and 8 
pixels are observed with blue again. Y- 2 pixels of no color are observed next. At time 3, the 
1 5 blue feature has moved to the right one more pixel and so X + 3 pixels of no color are observed 
followed by the 8 blue pixels followed by Y-3 pixels of no color. 

It can be appreciated that recording the color value of each pixel for each color would 
generate a large amount of data. Figure 5C illustrates how the data can be compressed by 
recording only the changes in the color data, and then using run length encoding for those 
20 changes. Run length encoding techniques are well known to those skilled in the art. At time 0, 
no color is observed, and so the data stored is simply no color or black. At time 1, X pixels are 
still no color, so that number of pixels is recorded and associated with no change. Next, 8 pixels 
have changed color to blue and so 8 pixels are recorded as having a positive blue change. 
Finally, the remaining Y pixels have not changed, so Y pixels, no change is recorded. At time 2, 
25 the X pixels on the left side which had no color have not changed and so X pixels are recorded as 
having no change. Since the blue image has shifted two pixels to the right, the next two pixels 
which were formerly blue now have no color. Both pixels are recorded as two negative blue 
pixels since the color change for them was negative blue representing the loss of blue color. The 
next 6 pixels were formerly blue at time 1 and remained blue at time 2 and so no change is 
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recorded for the next 6 pixels. The next two pixels were formerly not blue and have become 
blue. Therefore, the next two pixels are recorded as having positive blue value representing a 
positive blue change between time 2 and time 1 . The next Y-2 pixels were blank during time I , 
and remain blank during time 2, so no change is recorded for the next Y- 2 pixels. 
5 At time 3 the blue feature has moved one more pixel to the right. The first X+2 pixels in 

the linear array had no color at time 2 and still have no color at time 3. Therefore, no change is 
recorded for the first X+2 pixels. The next pixel was formerly blue but, since the blue feature 
has moved one pixel to the right, that pixel is now has no color. Therefore a negative blue color 
change is recorded for that pixel. The next 7 pixels were blue at time 2 and remain blue at time 3 . 
1 0 Therefore, no color change is recorded for those 7 pixels. The next pixel had no blue color at 
time 2 and has a blue color component at time 3. A color change of positive blue is therefore 
recorded for that pixel. Finally, the Y-l remaining pixels were formerly blank at time 2 and 
remain blank at time 3 and so no change is recorded for those Y-l pixels. 

From this example, it can be seen that as a surface feature moves across the pixels at 
1 5 successive times, it is possible to represent the image of the feature by recording only the changes 
in color values for each pixel at each successive time. Successive pixels which do not change 
color can be simply noted without the need to store three dimensions of color value for each 
individual pixel. This is only one example of a data compression scheme which is used in one 
embodiment of the present invention. Other compression techniques can be used as alternatives 
20 to or can be used in combination with this compression technique. It is only required that the data 
be stored in a format which will be accessible and expandable later on for processing of the two 
dimensional images to determine a three dimensional surface. Alternatively, if storage capacity is 
not a problem, data compression can be eliminated. 

In other embodiments of the invention, surface attributes other than color are derived and 
25 stored in connection with the three dimensional surface data. For example, specularity, 
emissivity, and absorption can be derived from the surface geometry and the orientation of the 
local surface in multiple views at multiple angles with respect to the surface. In certain 
embodiments, these attributes also aid in distinguishing trackable patches. 
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Figure 6 illustrates the process 352 that identifies the silhouette of the object in each 
image. The process starts at 600. Step 602 tests whether the process is finished with the images. 
If the processor is finished with the images, then the process terminates at step 604. If more 
images remain to be processed, step 606 applies a filter kernel to the image which is herein 
5 incorporated by reference. As is known to those skilled in the art, a filter kernel is a set of 
coefficients which is convolved with an image (an XY array of pixels). Certain filter kernels, 
such as the Canny and Sobel kernels, highlight edges by producing a high correlation at locations 
which represent edges in images. Edge detection using filter kernels is described in: "On 
Detecting Edges" by Naiwi, PAMI, Chapter 8, pp. 699-714; "Finding Lines and Edges in 
1 0 Images" by Canny, MIT Technical Report TR-720, 1984; "On Calibrating Computer Controlled 
Cameras for Perceiving 3D Scenes" by Sobel Artificial Intelligence, Vol. 5, pp. 185-198, 1974; 
and "A Comprehensive Approach to Edge Detection" by Canny, PAMI, Chapter 8, pp. 679-698, 
1986. By correlating the filter kernel with the image at all points, it is possible to determine the 
edge of the object for the object silhouette, by analyzing the correlation value. The silhouette 
1 5 edges are determined in step 608. Control is transferred back to step 602 to check whether there 
remain other images to be processed for silhouettes. 

The purpose of determining the silhouette of the image is to constrain the amount of area 
over which surface features are searched for and identified for the object. A flying insect or a 
noise spike recorded by the detector may exist outside the area of the object shown in a given 
20 image. Noise spikes or insects may appear to be a good candidate to a be a trackable surface 
feature, but if the silhouette or edge of the object has been determined and the insect or noise 
spike is outside of that edge, then it will not be designated as a possible trackable surface feature. 
This simplifies the search for surface features and reduces computer processing of the image data 
by limiting the area to be searched and also prevents extraneous surface feature points from being 
25 identified and mapped as surface points. In certain embodiments, the search for surface features 
is further limited to an area within the silhouette as is described below. 

Figure 7 illustrates further detail regarding process 608 of Figure 6 wherein silhouette 
edges are found along each scan line. The process starts at 700. In step 702, the processor 
begins at the left edge of the image and correlates the kernel with the pixels of the image to find a 

17 



8NS0OCID: <WO 9817970A1_I_> 



WO 98/17970 




PCT/US97/19331 



correlation maximum which corresponds to a potential left edge. . In step 704, the processor 
moves in from the right edge of the image and correlates the kernel with the image data to find 
potential right edges. In step 706, a heuristic algorithm is used to determine left and right edges 
of the object. In this step, edges are distinguished from bugs or blips by the nature of the edge. 
5 In one embodiment this is accomplished by distinguishing between the continuous nature of an 
edge verses the isolated spatial nature of a spec or blip. Any alternative heuristic algorithm which 
exists for determining edges in such instances may be used within the scope of the present 
invention. 

Figure 8 illustrates the process 354 of Figure 3B implemented to determine a set of 
1 0 trackable patches. The process starts at 800. Step 802 determines whether the processor is 
finished with all images. If the processor is finished, the process ends at step 804. If the 
processor is not finished, control is transferred to step 806 and the processor locates the vertical 
center line of the image being processed. Next, the processor steps through the image to look for 
trackable patches starting at the center of the image and searching outward. 

1 5 The size and dimensional nature of the trackable patches varies in different embodiments. 

In certain embodiments, trackable patches are searched for in individual scan lines, so that the 
trackable patches are one dimensional. In other embodiments, the trackable patches are searched 
for across several scan lines at once so that the patches are two dimensional. The size of the 
trackable patch searched for is fixed in some embodiments at a certain pixel area, for example 

20 10x10. In other embodiments, the area of the trackable patches is not predetermined and may 
vary in size for different areas of the image. This enables the system to be flexible and select 
trackable patches of an optimal size for the type of trackable features found on the object. In 
general, when an "area" is referred to as being checked for trackable patches, any size or 
dimension area is intended, including a line of pixels. 

25 Step 808 checks whether the processor is finished testing all parts of the image. If the 

processor is finished, then control is transferred back to step 802. If the processor is not 
finished, then the next unsearched area of the image is searched and analyzed to determine if it 
contains any trackable patches in step 810. 
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A number of ways exist for determining whether an area or section of the image is a good 
candidate for a trackable patch. A good trackable patch is a series of pixels which have RGB 
vector values which are readily identifiable and distinguishable from the RGB vector values in 
other areas of the image. Therefore, a distinctive coloration of a patch is an asset to its 
5 trackability. The trackable patch RGB vector values will be correlated with the RGB values in 
other images which represent other views of the object. If the trackable patch is readily 
distinguishable, then the correlation will yield a distinct spike at the new location of the trackable 
patch in the other images and the patch will be successfully found or tracked. 

A number of methods exist for determining how unique or identifiable a given area in an 
10 image is, and therefore whether the area is a good candidate to be designated as a trackable patch. 
One method is to compute the variance values of the pixels within the area. A high variance value 
will indicate that the pixel values are changing and thus will at least be readily distinguishable 
from a solid or unchanging color background. A more sophisticated method is to convolve 
different kernels with the image area being tested to judge the quality of the surface features and 
15 its likelihood of being trackable. Standard kernels such as those defined in Canny and Sobel, 
supra, can be defined as a standard kernel used to check for trackable patches. Alternatively, a 
learning or smart system in some embodiments has the capability of developing customized 
kernels on the fly which are optimized for the type of surface features found on the object being 
scanned. The size of trackable patches, and the kernels used to identify them may be held 
20 constant or may be varied. 

As noted above, step 806 locates the vertical center line of the image. The system begins 
at the vertical center line of the image and the search proceeds outward for trackable patches. The 
reason that trackable patches are first looked for at the center line of the image is that surface 
features are most likely to be completely and accurately sensed without distortion when they aie 
25 observed head-on by the detector. However, areas to the left or right of the centerline can also be 
used. As surface features rotate away from the center line of the object, they will eventually pass 
out of the view of the detector. Before passing out of the view of the detector, the surface 
features will begin to become partially obscured and distorted as they are viewed obliquely by the 
detector. For example, at a 45 degree angle, a circle on the surface of the object will appear to the 
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detector to be an oval. Obscured and distorted surface features are more difficult to accurately 
track because they change between images and it is also difficult to define a consistent location 
point for them. 

In step 812, the processor marks the trackable patches found in the area searched. This 
5 includes storing the color data for the pixels in the patch and storing the location of the patch. In 
step 814, the processor increments a distance away from the center line to define a new area to 
search and control is transferred to step 808. As noted above, step 808 determines if the image is 
finished being tested. In one embodiment, this is done by checking whether the entire area within 
a maximum distance from the center has been searched. The distance from the center line over 
1 0 which the processor will search for trackable patches can also be varied. Figure 8A illustrates 
an image being searched for trackable patches by the process 810 of Figure 8. The top portion of 
Figure 8 illustrates object 100 and area 850 which is magnified below. Area 850 consists of a 
number of areas or sections 852 which, in turn, consists of pixels 854. Area 850 is located at the 
center line of object 100 and so will be tested by the processor on its first pass. Each section 852 

15 of area 850 will be checked to determine whether it is a good candidate for a trackable patch. 
The vector color data from the pixels 854 is checked to see if it contains characteristics which are 
expected to be readily identifiable and distinguishable from other areas. 

As noted above, the processor begins at the center line, C s and checks outward for good 
trackable patches because features are most likely to be identifiable at the center. In other 

20 embodiments, the computer can check for patches starting with a location other than the center of 
the image. In Figure 8A, the trackable patches are shown as squares which have a predetermined 
area or number of pixels, in this case, five by five. In other embodiments, the trackable patches 
have variable or irregular sized areas. Different predefined sized areas can also be used. The 
important feature is that patches on the surface are identified in the image so that the points 

25 corresponding to those patches can be determined and tracked in other images. 

Figure 9 illustrates the process 356 of Figure 3 implemented to determine the radius of the 
location of patches on the surface of the object as the object is rotating. As noted before, a set of 
two dimensional images of the object are acquired as the object rotates. These images include 
surface features of the object located in a two-dimensional space and it is desired to derive the 
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three-dimensional location of the features from the two-dimensional information in the images 
Features which are likely to be trackable are designated as trackable patches. 

A three dimensional surface is derived by locating each of the patches. This is done by 
tracking the patch across a number of views which represent angularly rotated images of the 
5 object. As the object is rotated (or virtually rotated if multiple detectors or a movable detector are 
used-whenever the object is described as being, rotated, it should be understood that virtual 
rotation is also included), features which are located a longer distance from the axis of rotation 
move a greater distance than features which are located a shorter distance from the axis of 
rotation. A radius from the axis of rotation of the object is derived for each trackable patch which 
1 0 will account for the movement of the trackable patch in the different images. 

The process starts at 900. Step 910 determines whether the processor is finished with all 
patches. If the processor is finished with all patches, then control is transferred to step 912 and 
the process ends. If more patches remain to be processed, then control is transferred to step 913 
and a patch is chosen, together with an initial image in which the patch is found. In step 914, the 
1 5 processor checks whether the patch lies outside of the tracking area within the image. If the patch 
is not outside the tracking limit, then the processor calculates the expected path of the patch in the 
image in step 916. If the image is the initial image to be processed, then the expected path will 
simply be the initial location of the patch in that image. 

Next, in step 918, the processor finds the exact position of the patch in the image and 
20 stores that position for that image. In step 920, the processor updates the patch kernel. The 
reason for updating the patch kernel is that the patch will change as it moves across the image 
from the center towards the tracking limit. The tracking limit defines the area of the image in 
which patches will be tracked. This area is bounded in one embodiment across a 90 degree 
range. Although the tracking limit is set so that features within the limit remain relatively 
25 constant, it is possible near the edges of limit that the image of the patch will change more and 
more rapidly with angular displacement. The changing of the image of the patch makes it difficult 
to track the patch and increases the possibility of error. 

Although within the region over which the patch is tracked, the patch may change 
considerably from the center of the region to the edge of the region, it is fortunate that, in many 
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cases, the patch will change in a relatively continuous manner. Therefore, although the patch 
when viewed at the edge of the patch tracking region will appear very different than the patch 
when viewed at the center of the patch tracking region, the patches viewed in areas of the patch 
tracking region which are close to each other will be similar. To take advantage of this, the kernel 
5 used to search for the patch is changed with each successive image so that the patch searched for 
in each image is actually the patch which was found in the last image, not the initial patch which 
was defined in the center image. This makes it much easier to correlate the patch and to 
successfully track the patch. 

Once the patch kernel is updated in step 920, the processor obtains the next image in step 
10 922 and control is transferred back to step 914. When step 914 determines that the image limit 
has been reached, control is transferred to step 940. In step 940, the data is filtered or smoothed. 
Filtering and smoothing consists of applying known standard statistical and filtering techniques 
for cleaning up the data and perhaps rejecting bad data points. The data consists of the positions 
determined across a plurality of images for each patch. This data is associated with the color 
1 5 pixel information for each patch. In step 942, a recursive least square ("RLS") algorithm is run 
to fit the points to a curve. In step 944, the radius from the axis of rotation of the object to the 
surface of the object at the trackable patch is determined which will account for the motion of the 
trackable path across the images. Control is then transferred to step 910 which proceeds to the 
next patch or, if all patches have been processed, control is transferred to step 912 and the 
20 process ends. v.* 

At the completion of the process 356, three-dimensional surface information about the 
object has been obtained. That three-dimensional surface information is integrally linked to color 
data because the vector color data of the trackable patch was used to find the patch in each 
successive image so that the motion of the patch could be determined as the object rotated in 
25 different images. The radius of the surface portion containing the trackable patch combined with 
the two-dimensional pixel information obtained from the optical detector is then be transformed to 
provide a three-dimensional representation of the point in any desired coordinate system. 

Figure 9A illustrates a set of patch tracking limits as used by step 914 of Figure 9. . 
Detector 1 10 views object 100 straight-on at point 900 on the surface of object 100, i.e. along the 
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centerline C t of the object. As noted above, surface features and trackable patches are viewed 
with the least amount of distortion at this head-on angle. As the trackable patch is rotated away 
from point 900, it appears to detector 110 to change shape and becomes distorted. In one 
embodiment, the judgment is made that it is not worthwhile to attempt to track trackable patches 
5 beyond point 902 and point 904, which are located at a 45 degree angular limit from point 900. 
The tracking limit in each image will be a vertical line corresponding to this angle. In other 
embodiments, other limits are chosen. Limits can also be learned by the system or determined 
based on the nature of the data so that no predetermined limits need to be set. 

Figure 9B illustrates the motion of trackable patches across different images with different 
10 angular displacements as calculated in step 916 of Figure 9. Object 100 in this image is a 
symmetrically shaped vase so that the silhouette of object 100 remains constant but features on 
the surface of object 100 move as object 100 is rotated. Object 100 is shown with a trackable 
patch 950, a trackable patch 952 and a trackable patch 954. Trackable patch 954 is shown located 
at the exact intersection of the horizontal and vertical center lines of the detector view. When an 

1 5 angularly displaced image of object 100 is analyzed to find trackable patch 954, it is not necessary 

to search the entire image for the trackable patch in a random manner. Trackable patch 954 is 
expected to move along a line 956 and so the search for trackable patch 954 in a successive image 
of object 100 should be executed along line 956. Note that line 956 is straight because trackable 
patch 954 is located at the center point of the detector view in the first image. Similarly, trackable 

2 0 patch 952 is expected to follow a line 958 in images in which trackable patch 952 is angularly 

displaced, and a line 959 is the path which trackable patch 950 is expected to follow in angularly 
displaced images. Line 958 is curved, unlike straight line 956, because line 958 does not pass 
through the center C d of the detector view. Features not located in the center are expected to 
follow epipolar trajectories as is known to those skilled in the art. Line 959 is even more curved 
25 than line 958 because line 959 is further from the center of the detector view than line 958. Thus, 
it is possible to quickly locate the trackable patches in angularly displaced images because the 
expected path of the trackable patch is known from the epipolar trajectories as object 100 rotates. 

The distance the trackable patch moves along its expected path for a given angular 
displacement, or the speed of movement of each trackable patch as the object rotates, is not 
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known. The change in position will vary with the radial distance of the trackable patch from the 
axis of rotation of the object. This is how the radius from the axis of rotation of the object is 
determined. 

Figure 9C illustrates the determination of an exact position of the patch in an image as 
5 determined by step 91 8 of Figure 9. Trackable patch 960 was observed in a former image and its 
location is defined by its center point 961. The expected path 962 of center point 961 is known. 
The processor searches for a good correlation of trackable patch 960 with the image data for 
patches along expected path 962. Patch 964, represented by a dashed line, does not present a 
good a correlation as patch 966, which is represented by a solid line. Patch 966 is determined to 

10 be the patch and the location of the trackable patch in the new image is determined to be point 968 
which represents the location of patch 966. 

In this example, the center point of the patch is used to represent the location of the entire 
patch. This approach works as long as the patch is small. In other embodiments, multiple points 
are defined for a patch or a weighted average point which is based on the colors of the pixels 

1 5 which make up the patch is defined for the patch. In the illustrated embodiment, the location of 
the patch in the different images and the motion of the patch in the different images which are 
angularly displaced is represented by the motion of the center point 961 which represents the 
center of trackable patch 960 to point 968 which represents the center of the angularly displaced 
patch 966. 

20 Figure 9D illustrates the filtering of the raw data points as performed in step 940 of Figure 

9. Point 970 is shown as being inaccurately derived because it is located away from the expected 
curve. After the filtering process, point 970 has been moved to point 972 which lies in the 
expected path. A number of standard filtering techniques can be used. The frequency of the 
rotation of the object is known and all surface feature motion should occur at a spatial frequency 

25 corresponding to that frequency. In one embodiment, a brick wall filter is used to filter the data 
to remove motion occurring at other frequencies. 

In embodiments in which the object is not rotated at a constant frequency, or where views 
of the object are obtained without rotating the object, (for example, by using multiple cameras) 
then the spatial frequency of the occurrence of features in images will not correspond to a 
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frequency of rotation of the object. In such embodiments, the raw data is filtered using a 
modified Z-transform method. Modified Z-transform methods are used to filter data when the 
displacement is non-uniform. 

Figure 9E illustrates how the radius is determined from the points representing the path of 
5 the trackable patch across angularly displaced images and corresponds to steps 940 and 942 of 
Figure 9. Figure 9E is a plot of curve 980 which represents the first derivative of the distance, or 
the speed of the patch as it is angularly displaced from the center of the image versus the angular 
displacement of the patch from the center. When the angular displacement is zero, the 
displacement of the trackable patch is zero and its linear speed is a maximum value. As the 
1 0 angular displacement increases, the speed will change according to the cosine of the angle of 
displacement. 

The radial distance, R, of the patch from the axis of rotation of the object can be 
determined from a graph of the speed of the patch as shown on curve 980. Note that the 
frequency of this curve is well known so the data can be filtered and the amplitude of the curve 
1 5 can be accurately determined. After fitting the actual rate of displacement of the trackable patch 
to the cosine curve, the amplitude of the cosine curve is determined. The radius of the trackable 
patch is derived from the amplitude. Other processing methods are used in different 
embodiments to derive the radius. In one embodiment, the displacement of the patch, not the 
velocity is fit to a sine curve. An embodiment can use any processing method available to derive 
20 a radius which accounts for the patch motion. 

Figure 10 illustrates the post processing process that can occurs once the radius of the 
trackable patch is known. The process begins at step 1000. In step 1010, the radial distance data 
for the trackable patches is combined with the two-dimensional trackable patch data from the 
images and converted to a desired coordinate system. In step 1020, the color data associated with 
25 the trackable patch is converted to a desired color system for storage. Step 1030 performs 
interpolation and decimation of the points. In one embodiment, all of the trackable patch points 
are interpolated so that a regular array of surface points can be defined to model the surface of the 
object. In step 1040, the points derived in step 1030 and the color values associated with those 
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points in the form determined by step 1020 are stored in a data structure. The process is 
completed at 105. 

In this process, it is especially important to note that no matching of color data with three- 
dimensional position data is ever required. The color data for the trackable patches is already 
5 associated with the trackable patches which define the surface points of the object, and, in fact 
was used as an aid in calculating the three dimensional locations of those points. When the 
trackable patch points are decimated and interpolated, the color values are likewise interpolated. 
No misregistration of color occurs because the color information is already associated with the 
patches whose locations are determined to model the surface of the object. 

1 0 While this invention has been described in terms of several preferred embodiments, there 

are alterations, permutations, and equivalents which fall within the scope of this invention. It 
should also be noted that there are many alternative ways of implementing both the process and 
apparatus of the present invention. It is therefore intended that the following appended claims be 
interpreted as including all such alterations, permutations, and equivalents as fall within the true 

1 5 spirit and scope of the present invention. 
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Claims 



I . A three dimensional digital scanner comprising: 

multiple view detector means responsive to a broad spectrum of visible light, said multiple 
5 view detector means being operative to develop a plurality of images of a three dimensional object 
to be scanned, said plurality of images being taken from a plurality of relative angles with respect 
to said object, said plurality of images depicting a plurality of surface portions of said object to be 
scanned; and 

a digital processor including a computational unit, said digital processor being coupled to 
1 0 said detector means, said digital processor being responsive to said plurality of images and 
operative to develop with said computational unit 3-D coordinate positions and related image 
information of said plurality of surface portions of said object; 

whereby a three dimensional image of said object to be scanned can be developed by said 
digital processor that includes both shape and surface image. 

15 2. The apparatus of claim 1 wherein said multiple view detector means includes at 

least one detector and means for rotating said object. 

3. The apparatus of claim 1 wherein said digital processor is operative to determine 
the boundary of the object in at least one of said plurality of images. 

4. The apparatus of claim 3 wherein said digital processor is operative to determine 
20 the boundary of the object in at least one of said plurality of images using a mathematical 

function. 

5. The apparatus of claim 4 wherein said mathematical function is a correlation 
function of said image information with a predetermined kernel. 

6. The apparatus of claim 1 wherein said digital processor is operative to identify a 
25 trackable surface feature within at least one of said plurality of images. 
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7. The apparatus of claim 6 wherein said digital processor is operative to determine 
two-dimensional positions of said trackable surface features within at least two of said plurality of 
images. 

8. The apparatus of claim 7 wherein said digital processor is operative to determine a 
5 three dimensional position of said trackable surface feature from a difference between said two 

dimensional positions. 

9. The apparatus of claim 1 wherein said detector means includes a color detector, 
and wherein said related image information includes color. 

1 0. The apparatus of claim 6 wherein said digital processor is operative to determine a 
1 0 radius from a trackable surface feature to an axis of rotation. 

11. A three dimensional color digital scanner comprising: 

a color detector responsive to a broad spectrum of visible light to develop a plurality of 
images of a three dimensional object to be scanned; 

a rotary object support having an axis of rotation, said object support allowing said 
1 5 detector to develop a plurality of images of a three dimensional object supported by said object 
support, where said plurality of images depict a plurality of surface portions of said object; 

a digital computer coupled to said detector and responsive to said plurality of images, said 
digital computer tracking patches of said surface portions of said object to determine coordinates 
of said patches as a function of the rotation of said rotary object support, said digital computer 
20 further determining radii of said patches from said axis of rotation. 

12 The three dimensional color digital scanner of claim 11 wherein said digital 
computer is operative to determine the two dimensional boundary of said object within at least 
one of said plurality of images. 

13. The three dimensional color digital scanner of claim 1 1 wherein determining the 
25 radii of said patches includes determining a first derivative. 
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1 4. A method for scanning a three dimensional object comprising: 
developing a plurality of images of a three dimensional object to be scanned, said plurality 
of images being taken from a plurality of relative angles with respect to said object, said plurality 
of images depicting a plurality of surface portions of said object to be scanned; and 
5 computing 3-D coordinate positions and related image information of said plurality of 

surface portions of said object from said plurality of images such that a three dimensional image 
of said object to be scanned can be developed that includes both shape and surface image 
information. 

15. The method of claim 14 wherein computing 3-D coordinate positions and related 
1 0 image information of said plurality of surface portions includes determining a radius of at least 

one of said surface portions from an axis of rotation. 

1 6. The method of claim 14 wherein computing 3-D coordinate positions and related 
image information of said plurality of surface portions includes determining a tracking limit for 
the tracking of said plurality of surface portions. 

1 5 17. The method of claim 14 wherein computing 3-D coordinate positions and related 

image information of said plurality of surface portions includes correlating a kernel representing 
an image of at least one surface portion in at least one image with an image of said surface portion 
in at least one other image. 

18. A method for determining three dimensional coordinates of a surface portion of an 
2 0 object comprising: 

obtaining a plurality of images of said surface portion of said object; 

identifying a trackable patch of said surface portion in an initial image of said plurality of 

images; 

determining an initial set of two dimensional coordinates of said trackable patch in said 
25 initial image; 
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analyzing said plurality of images to determine at least, one additional set of two 
dimensional coordinates of said trackable patch in at least one other of said plurality of images; 

determining a radial coordinate of said trackable patch based on said sets of two 
dimensional coordinates; and 

5 determining a set of three dimensional coordinates of said trackable patch from said radial 

coordinate of said trackable patch. 

19. The method of claim 18 wherein analyzing said plurality of images to determine at 
least one additional set of two dimensional coordinates of said trackable patch in at least one other 
of said plurality of images includes a correlation. 

1 0 20 * The method of claim 18 wherein determining a radial coordinate of said trackable 

patch based on said sets of two dimensional coordinates includes determining a radial speed of 
said trackable patch across said plurality of images. 

21. A method for determining three dimensional coordinates of a surface portion of an 
object comprising: 

1 5 rotating said object about an axis of rotation; 

obtaining a plurality of images of said surface portion of said object as said object is 
rotated about said axis of rotation; 

identifying a trackable patch within said surface portion of said object; 

determining the two dimensional coordinates of said trackable patch within said images; 

2 0 tracking the movement of said trackable patch as a function of the rotation of said object; 

determining a radial distance of said trackable patch from said axis of rotation based on 
the movement of said trackable patch as a function of said rotation of said object; and 
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determining said three dimensional coordinates of said surface portion of said object from 
said coordinates of said trackable patch and said radial distance of said trackable patch from said 
axis of rotation. 

22. The method of claim 21 wherein tracking the movement of said trackable patch as 
5 a function of the rotation of said object includes correlating an image of said trackable patch with 
portions of an image of said object. 

23 The method of claim 21 wherein determining a radial distance of said trackable 
patch from said axis of rotation based on the movement of said trackable patch includes 
determining the radial speed of said trackable patch. 
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15 triWfigt&So 

W ± . R ft) tff ? -fe/P22BH 2 5 l &<E>£ LtSUlLfc^, |(£ £ * ^ -tr /u^Bfl £ m 

20 (f^^ft^i'tV^) 

yofcWi&A l - Anli^iti5T^7,ftM$: kiTci 3 HkjtBtk* T' >v 3 OOl: 
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mmmimm) *nmi%m-zmi£m\&w>2 20 mmmm^2 2 o^&iti 

20 210^ 7fiHgP2 1 0 l-J; ^^n/i^^e^'/^ 0 0 , J: 9 &ft;ffjKLtt& 

OAK =^2 7{C*tLTfiJ^T/i^^^^-vt^^Hattf^-r5fe0t^ 

IEtESE2 4 0£{ira;l3„ 

25 Hi 3*3J:ai§l4^#BaLT, HtjiiL/it^i!}, *m4W£-&ft 

(^f y^S 20) „ 

^^t, *J&3frfciiHft1&«teS^-t\ ft£8l!2 1 Oj&s. ^^7=^3 00 
£tfi5cU *<DMVt J r — *&BVttQ.mi&2 3 0 M 5 (7,r^S2 2) 0 
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<t mmmu 220^ ^/c ^w.mmrm t omi&m-t zmtm 2 1 o & 
ftte? k^T-yrs 24) „ 

5 tS*KlEt»2 4 O (^#£-£3 (Xry^S 2 6) 0 

m 1 4 li , ^fy-/S2 4i:m4 ^ttWfifelf 45 <t # Bg iiifftffifti £ to 
fm«rff^9fc»<7)7o — S: «fc »9f¥L< ?nt7n.-ft- hT£>5 0 

15 iliM^ Ufs'7 , S2 0) ^.tt^lfc^/Wfr^ (^f7^S2 2) 

JO V> h-tZ>tzfr(DffiWm%i; lent *flt0t^JJ«fk-r5 o 

SkK, i#i (i=0-Emax-l) 
i"S 1 ^Tufil^J^^cF rod [ i ] Ofi£&i-'<Tfi£0(^;jig)-(fcL, — i §0 

» I d C i ] -T'<Xm- l {C&WHtZnZ (X^ yZfS 2 4 0 2) 0 

Wmtlrh {*7-vZfS 2 4 0 3) 0 

(^f^T'S 2 4 0 4) „ 
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fccV^T, /&52SR2 1011 Rent #fc|0O^&P rod [Kent] (O^.t ±. 

rd^tfev t mpcofiet^ri-b^-rao 

Prod [Kent] ^fiR^^IJcV t m p OH&VX YT" foZ> <bMWhf"5 t 
(^T^S 2 4 0 6) , ag»P rod [ E c n 1 ] lC'£&V t m p <Z>fc#ftA 
RB#(C^icl d [Ec n t] lC^ : C0D$,f.;T'<7)*'>> r-'£3& I c n t Ofcjfi 
ftA£;*X<2> (*7^>7°S 2 4 0 8) „ 

-15. $i%CP rod [E c n t ] <D{&?/\ t mp^jftj; V) kA^i^M 

•»fi"5 £ (^7=" * 7°S 2 4 0 6 ) x '£|&E c n t <7)fi£^ 1 fctt^ > • * V * > h $ 
(^7=- 2 4 10) 0 

£^ ft 5 <t (^7/S24 12) , SU=^T ,V7"S 2 4 0 4 I^Jfl/jS 

& £ (^T77'S24 1 2) „ # ? > h lent C0fijfj&« W > ^7 U ^ > h 
$;rX& (^T'^S2 4 1 4 ) „ 

lent (Dm^^mi^^H I m a x J£*±T*&<5 frEfrO) 

nm/>mtei?frz> Uf^s2 4 1 e) „ 

'^ifc lent (Dm^WLBmHk^: I m a x «fc K> ii'l^ts ^ £ PmtSiXtztM^ 
^^S24 16) , ^<D#B^jlJ^{COV>r, ^7-yyS 2 4 0 3A>^f77'S 
24 1 2 * W&JijM*ig3;h,5££t£fc<5o 

— lent <Z>«S|tE2iiif£tfc I m a x £X±.Vfr>Z> k¥*\Wi&iX : 5> <b 
T^S2416) , ^^A3.a^fT-t--5 0 

^r-yT'S 24 0 2frh^"rvzfS 24 1 6 £ -CCOAQ!?!^ cfc 9^ & 

rod [E c n t] {d*&#}£;H3 0 — 2r\ ^^"^^^Jlt 6#B8p{f^ 7-</U# 
^ 1 &7clid*l]'£ifc Id [E c n t ] *L5 0 
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■AW^JftfMSr^-r S0l!«ji!ii«iWf«<75 v^/Uffr J3-^ftiN«l$ tiro v6 - <t KjfcSc 
fBtM2 4 OKf&fcH-S (Xf-^S 2 4 18),, 

m 1 5 li, pel 1 4 {d>f< L ybr- ? * ^ \ tt5--JjUxZ ^ > 9 1 30 \z.%Vf £ 

* - y - k*-> y * * y ) * ifoyym^j * y / £ t'&m^z - £ ^iro-cfes, 

^±cD«t 9CLT, fti«8l(fc2 6 Ort'JCgii^^tL/cx^X^i-t^f.j-^^py 

mm<Dmm 2 ] 
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M-J-^Mgfl 2 o 0(Dffitttm$£Tfo?> o aTHIUntfcO, ^»SfR2 1 0 

-r&fr*>, r&ss^s i -cis, =*>2 7(co^t, ^oyfeu^v v^t 

20 Tli, EI 1 4 |:^Lfc7n-f + - hi^:< [5]#|-C£>5„ 

Lfc/jSoT, Xf77°S 2 4 2 2^b^f7 7*S 2 4 3 6* TO^l^TL 

r e a [ i ] Kfi, *fJ&-T£ i #g©iWMHil:o^T, Id [ i ] (7) 

^r'tlc, WJ©i-5#BSii5«<Z>7 i ^^^^««S:jii«get6»IJ2 2 0^f ) iffiLT 
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t^#!Kilifiitttff^/)'0^?K y 2 7 ^cOv 1 ? * ^ tfi#&f !-'->. -J- £ - £ ,yi .iffts <j- 
# 5„ £ /c „ |2l 1 6 C^LfcJ; ') 4 % 7>f 7 _ /S 2 4 2 L^^jXrv/S 2 4 3 8 

# V *>$Lftm& H5oTLJt\ WM^h/h into*: t*>v 2 9 ttfli tf> x ^ ft & 
B8iiitt^8iR^£fc*Kte % ttJ&i-^fM&^co^ y ^>«0ttJ5gffittiiA-£^ 
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tz ft , li|g#^#fMf£« £ L T*J V =f> 2 7 t #08PIfM ^ £ 

E= 23 Penalty(i)- k x area(i) ••• (l) 

i 

10 ::t\ Area (i) tttf? V =f> i <D0mmi%^<Dt9:&mn&2tt> U Pe 

n a l t y ( i ) ttsKU =*> i t^^m&tfV t eO#RgiBjflM& 

25 ir-TS ~ t fc^IfiBTfcSo 

£<bl^ ^*/U3rM#E£ LTIi. ±i$<Dj;';WPena 1 t y ( i ) t 
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f»Area (i) <t ^ff^i^-J^M- i> , J; «9 J&tol^ f-K R -f -5 Jl* 'J =f> i 

Wiiiift#^-^^^Mtt^fr*9 (xf 7 ^s 2 4 4 0) ,. 

10 Cn t £fMlffO{c:#JSMrf 5 C ^7^fcF 1 gf; "OFF" fcftA-fS 

(7,7=-.y7°S 2 4 4 2) „ 

«^T, lESPre. IblCCnt & B W A^^WfiJc^ld^iCi- 5 &Hfl 

4 6) „ 

"T* y/S 2 4 4 6 t^^3V^~C^ &i^,/^^f L V ^^tx£;^nSeifftl&-^-^^fl£fi: 

N e w__ I b 1 KftA-TS (^^S 2 4 4 8) „ 
Ife^T, ^tNew_l b 1 tftP r e_| b 1 t £ ft£„ |Bj# 

9, 9^W^t»^^tf**?ft/S:i:¥llWf$ft % 7 J g(c "ON" ifi 

RA£ft£ (xf-^S 2 4 5 2) „ m^X, K'^iScC n t eofft^s l fc'tt 

y h£ft<5 (7fyrs 2 4 5 4) 0 
— ^f, ^tfcN e w_ I b l £i»P r e_ l b l <7>ftt/j^ L V >Jt-£H2, 77/ 
25 |g|fc«^fb$ft-r{-, 7> hf£ffcC n t <7>jj£co<?^5 l t£\-tJ^9 V * > h £ft 

5 (^xs/^S 2 4 5 4) „ 

SU £ «! ! fflAS^7 t y"7'S 2 4 4 4lcMH-3o # ? > n t <OW 
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56) 0 

Lfc^oT, ^T y^S 2 4 4 4 fat,*^ y^S 2 4 5 4 it'OJUlI/j!, 1~-< 

fce^T, 7 7»FI g t "OFF" t <?>tt&Wi tefrtlZ* ^^'«F 
5 lg/j5"OFF" ^L<4H^tt (7f77'S 2 4 5 8) , '>^<irtl|iil 

^^ir-Wirr$n> ^JPJifW^^S 2 4 4 2l:Iit5„ 

77^1lcFl g/j* "OFF" ir^LV^J^fi, y7"S 2 4 4 4^ 
10 io^-r.y^S 2 4 5 6 $.X*<nmmiZ.&\,^T. zi*/l'*cM } &E$;M>Mt-fZ ) 1%%$: 

Jl7)Sif&T1-5 (^^^7°S 2 4 6 0) „ 

iifk-r a x ^ * ^ mm <D-m % /^tT^^s r t \z. & 6„ 

±j|<D^T7/S 2 4 4 Ofrb^'r V7"S 2 4 6 O^-Ctf^n r^^rfg^L 
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J- £ n /c I£ CO fftifiSf jjs /J ^ $ p ;K 'J =f > M £ (£ ^ X' 3 ,. 
# V =r> - »c t 1 ? * ^ -r tff « fctt-^-f 3 'Jjfe «r&{ft--f £ „ 

<£><5o 

IKlT-§B«1-5»-g'S:%^.6 0 &*«<7>T£,6£r (xO, yO) , (x 1, y 1) , 

(x2, y 2) , #naiai««a^^LT«j^$nfe^*^w^^<7>iH^^ 
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10 



15 



20 



(X0. Y0) , (XI, Yl) , (X2, Y2) ft2>< 'K^^r, UTRC^ 



-tf)4&£\ 2£*J0*fcWii5SS (xn, yn) <£>x^ *?--rtfH»tt, J^'F^itT'iil 

if $n5#!Hiiiifei*«.h^w* (xn, Yn) fabmn-rtitzx^z t&n#i-r 

<5 0 



ffi»ie«*P 2 4 0 |C^f5o 

_ ajx n +a 2 y n +a 3 _ a 4 Xn -t-a 5 y n +a 6 

A n ; ~V' Y n T " ' 4 > 

a 7 x n +a 8 x n +l a 7 x n +a 8 x n +l 

tt £ <t o fsj -t £ -it s ?■ 9 * f- -r tf m ^ & & gig -r 6 c t % n & t -t s 0 




•• (2) 




- (3) 
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t ^wo(c^j9ii3:ze$na (^f-y^s 2500) „ 

m^X, h&%t lent ftfjUtC^w a c c Wfi/jSo (C^Jjtq^^tL 

-5 (^yT/S 2 5 0 1) „ 

2) „ 

h t id^tLT, 1 c n \.#f\<D&Mmmfttn\z.$(te&frizE c 
n tmn<?)K{*Bmnf&mm<Dm&(DmyMtA£Hz> (7f^s 2 5 0 4) 0 

M^T, Ren t#B^.u^Bt»:til^*«7 i ^^^A't^^i LT, lent 

&52 4 0lC&#rf5 (7sry/S 2 5 0 6) 0 

£»wa c c I^LT, f|[wg h t omZmmL (X^ryzTS 2 5 0 8) 

c n tcOM^l (^s/^S 250 

9) o 

(xT>y7S 2510) „ 

ggftl cn ^fti ma x J; >9 t>/h£lWJIl^ */ijg#, SU^f^^S 

2 5 0 2^^-r^o 
-ft, ^T^S 2 5 0 2 tC*5V>T, lent #H^$!g|iH%ffi$B;aS % E c n 
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jcmfrxr- y"7S 2 5 0 9 Id^ff U $ztk lent <Oti£.ft> 1 titfjis'/ 'J ^ y k 
£H/cf£ (^r-y^S 2 5 0 9) , lent I m a x 

Lfc^ot, ^xs/^S 2 5 0 O^b^fy^S 2 5 1 0 * T'60*G.Jij4Stfejg£ 

m^x. &m%miimu2 a o ^mmztitz^ ? x^+rafiw^sw a c c o 

fi-e#l^^n5 (^fy^S 2 5 1 2) 0 

io criTOMt-, Ecn t#@roWtMI$-W5f^^ftii 

^M^Ifmt-^-raS^lfe^wg h t (lent) £1-S U *Ac; 

25 ^rT% &&£HtlMitL<Dmi$LWWj> v (I cn t) xfo^tzt-rz 

[X wght(Icnt)xv(Icnt)]/X wght(Icnt) ••• (5) 

iBc^T. '£ffcE c n t (Dm& 1 fcttl' V^D^y h £*x£ (^fy7*S 2 5 1 
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4) o 

^ , # > H E c n t COM £ £f*^K«/#3g#» E m a x t eOfiff it 

(^-r-y^S 2 5 16) 0 

^^y^S 2 5 0 1 |CijSf7,„ Lfc*ioT, t'<TWiM^f^||^|; 0 
-A", h«E c n t /j^ uWIM^ItEma x ^±Tfo54»^ 

SsWr-TS (^f7^2 5 1 8) „ 

fab mm l fc # Rgmift is t c *j it 5 a m x v =*• > k at-t- 5 tt m k £ * *l a # 

01 9 iZ.ijkLti <fc 0*£*V- ^^S 2 5 0 Ort^^x y^S 2 5 1 8£-CcOT"n 
[»<DM?5] 
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& ?K 'J ^> K # L T t 1 * * ^ -r 84 & - x. -5 ti? J& I d o v * T fft H/l L Ac„ 

m&(nMB5 0)^? x^^ffintt^-X&viz. fr#y=f>> {iLfoi&mfir&m 

tt-? x=?-i>mm*tf5-LTt,&\<^L, fr#y=f^ (<Li*Mmftj&wm) 

A » P, ^ g£ ;K y =f > K *t L T T 9 * ^ -Y ffi « £ f t "T 5 # H & ^ t T V % 5 „ 

r^S22) #£fr7F^tfffi£^i?f SKiE«c^«fifcl?^roSJ^a5IB 

*soT/j;v^Rgi5«fflf«i:oWjc£:f+tt^tf*5 (^f^S3 0) 0 
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x<Dfflm-tz&wmm%wt»tb<DTj' *?- j rffim&m?*¥-&)i-z>ztx\ z<d& 

3 0£^;t5 o ^yfa-^i 3 011 C D— ROM 2 6 0 Idufi^^tlT'' ^ 
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■Sr^t^ny^0t?*)S, H12 3 ^.iLT, «MMAttll£il <t L T 0) =i is fa - 

9 1 3 cm, pifttaitsp2 2 o, ^Mi2 1 o, m®\nmm$r>2 4 1 *j«tut«j 

tiAMfAfctiiB2 3 i <e*3, ftg|S2 20, 210, 2 3 1 , 2 4 1 <r>n 

l*]2 4»:t, HI 2 3<aiftftft|S2 1 Oc^BftSr^-T^n y ?mT*}bZ> 0 [g| 2 4 Sr# 
RBL-C, iH3?S&2 l oil, «J«^>lSi|«J9*jJ:TjtfthtflSFIJi 0£tf§;t3 c ffctt'.gin 0 

M^&Jigfn 1 x sp*&«ranugin 3 S LZ^mmmtfn s&xvtmftmftfa 
0 l^r^^s 1 ) 0 miti&ymzm-tztnm-z, m2 3^«itM2 a 

TOfjS^r^ttJi-^ (fc*]2 2<Z>?°c7^ J k3 O 1 COXt^^^S 4) „ 0*19, spj^j 
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fl\ rm*mi* SWo plus E, iHf^Mryu=f!J *-kwftfcrWH&h p 
p. 227-233, SftSW.n * a-4r-*>a >Xj {cfTC $ ixT V>3 J: y (C, - 
AJMCffl^fcftT^*, ^y^&Sfe, fiSJ* • ^-y^ffljSfe, Facet mod 

|gl 2 5 Afahm 2 5 CH, l*| 2 4 W^ftigifflS 1 1 , Wj^mSlS 13, L*- 

v mrs^m 1 5 jr3 x umwdmbu i 6 iz&i-t zmm&& l < $twtz>fc#><7> 

ftfciiHft 2 3 tt\ 2 5 3 „ Vvitmm l 7(1, &I2 4 

©««»t«l*BJ9(c:J:o-c. m&emWa i~ a nl:^|$jitiv5 0 

Mi&a ICttiLT, fel 2 4 <Z)£g##lUPJP 1 1 wBMfadoV ^T!#P/J-f S„ @2 
5 B t#RR LT, fiSJsJc a 1 0#ifi|^fetf «i , fSfc£ a 1 K*tJ&-f -5 2 5 W 
fPJ&b 1 OM^<Z)fetS$8£cD|£&;fc#>5 0 ~?>J;9lcLT, mJIca 1 (c*S(t£ 
c l£*#5o -t L~C, Ed2 4«^«[i»IH*Bl 3fl, mtt<D$k&c 1 

4<aL#vH£«tg!fiBl 5(1 M^m^-c 1 Srffi^-rs^wlfe^fio^fifft^ 

*fc!i*a 1 Sr^flsSP^i LTmttJi-5„ tg#*Q.JaS» l 1, ^fit^ttj^ 1 3*5 J; 
V L •# V MiS&lSgfl 1 511, -t-'<TG0«HJ«a l- a n(:oi>r, Jiidi L tz. J: o ft & 
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warn, mftnmttm&mmmm&xxfiLZb^m&m&ftte'j,, tit, w 

SMMillii® i 6 11, L£ v ^M^^Jims l 5 C J: oTffc.'JJ £H/c#jfo«>&. »Pi 

&(D mm \z.^m l . mm mx. x*m&<D&*tm<o w-^m w-mfci* l % v 

emBHfe-CifcoTfcJX * ±i£Lfc&tt$ (feiH^-i^/W l:ft^tM : 
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ft-*- 3 T^zWdjfc C Tf£3I L T *> «fc 

20 mi&(7>&m7] 

m 2 6 is, *&m<nmn<oMm i tc j: s^wittj^ig^^fig^-t-^o- ■ 

— h"Cfc5„ 2 6^f-^S 112^3 11811, nyt'i-^130 

ftZmmZitZtzlbCD-T'ti KXh^X , CD-ROM2 6 0|C^^nt^ 
25 £ 0 

:ro^y 7 Ait ^-^s i i i x&btitz®frm&<7>$Lftffiwd p ( i , 

j) £r#iliJfi ( i , j) rtl^f^fticiotWtHtSxfyT's l l 2t> 

P (R) ?r^Rrti:|ltlit5xf^Sl 1 4h *fTl**0¥JS«m d 
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p (R) ^toZgfefbnfc L£HSd p t h tKMrTZXrvT'S 1 1 5 t, J& 
ftffiWco^mfcmd p (R) /^L£v^£d pthJ:*) i>-X$\<^M'frMvm!&R 

\ZT ^b^TVfS 1 1 6 t, %ft\n®L<D¥-®mmd p (R) fl* L# l/MSd p t h 

<^#if51^<^ffi v ( i , j) S:l|:t57f *y/Sl 1 7h ±E^f y^Sl 1 
5~S 1 1 7 (D&m&i£.t&&Rl^\,^T'ttte^tcfa : &t)>$:W£-fZ ^"r y7°S 1 

h olfl^/j: if ^ 3 - £ ^ t- £ -5 o 

0 rt©iiiflfetet6Sf|J 2 2 0l£f&#j£H5o dHKJ:>9, (i, j) (D\fet L 

15 Tv (i, j) ;Wb*L5„ -w-Cf±, tfritm*:ti§L&-rZ>7^J\'l> * ^£rfflV^S 

^>t^T5//si 1 2{c*j^-t: v *^i^*fc#£*{^oT£ili=lt (i, j) 

<0!!fTtit«d p . ( i , j) ^^mi-^o ^-rl/^&fi, it <b;ttf D.H.Ballard 
20 flb^ fnyfcV-^f^gyj , Prentice Hall , pp. 8 8 — 9 3 (dflfl^ $ 

^^ffl0»lfi/ctx.tf1t^^8-3 3 1 6 0 7^^{^P^7it$tLTV^-5„ **3, 
(Shape-from-Motion ) % miHtt*5«ttJ«iilKI±«: t LfefifeiS LflfeSF 

*x^:/s 1 1 2 b&fiLXX7-?7°s 1 1 3 d&^Tfi, ISJg$ftfc4fcft:iffli 

45 

BNSOOCID: <WO 9809253A1 J_> 



WO 98/09253 




PCT/JP97/02997 



»>f-^«mdp (R) £im!i-<5 0 



(R) asL&iH&dp t h £fcfct&S*L£ 0 ;roLt^«dp t h lifc?^ 
*frfS«0>>P*&t£md p (R) #>L£vMiiLd P t h X "9 *>*:ti*lg£\ ^o-ffl 

^Rp^(7)p^(7)^ v (i, j) /jw-<T0(c$n^ t , -r^-^ ^{M^^ 

L^V^ftd p t h ct "9 *>>h$V^§3\ ^fy7*S 1 1 7 (C^T^GDfM Rf*)(7) 

i^TXry/S 1 1 8{C*dV>T, ±rd^x^7"S 1 1 5~S 1 1 7c^)^i®^ 
t-^T<DM^Rl-o^-CfT^^nfc^^7)^^JgiJ$ti5 0 ±§BM/jH--<TW® 

mm r r t izman m <d w-^m. ml, ^ <o m%m& ^ z> & 46 h w- l # v > 



mdp(R) = 



Lr d P(U) 



■:• (6) 



n 
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5 f + --ht?*)5 8 BI2 8W^T, ^fy^S 2 2 2, S 2 2 4-S 2 2 7I1 

-ROM2 6 0l:E»$jlT^5 0 
10 IcynOAU, 2 2 1 (w*J^T^<bK/c^t!(^W^:ili^tC*3V > 

TS:^(;JwiDM^(;fi^-rsiiimco3pi$jMm ( i , j) JaimiTOo (i, 
j) Sr#ii5*rt -y^S 2 2 2 Xr^S 2 2 3|;fct^l 

bti1zVs{£mi&<?>4rm£<Dl&v (i, j) t ^cojSj^fci^i-^tf^iSj^^iij* 

wspMMBm (i, j) t(om<Dtettm. (jwt, mic r^j ) I v (i, 
15 j ) -m ( i , j ) i £»mu *-<z>si i v c i , j) -m (i, j) i &mm 

(Hlla(i, j) Ok{Silt^t57vf-^S 2 24t. II v (i, j)-m 
(i, j) MSko (i, j) «fc <9 hKZ^^&^nmmZlgfkUftk L-Cil'J 
-t-^^^^WlH^^jSv (i, j) ^OIC-TS*^ .y^S 2 2 5 t, gE I v 
(i, j) -m (i, j) MSkir (i, j) iH^tV^I^WB*** 

20 ftnfttLxmm. -ttei?h"t<nmm<Dmv n, j) ^n:t5^f-y7 , s2 

26t, -Lga^^s/T'S 2 2 4 ~S 2 2 6 cD#yg£:-r'<Tc7:>jIi^f |::oV ^Tff * o 
t^^W^t^^fy/S 2 2 7 ££^ir 0 

C<7>Hl£^168^J:5«5ft:ftbm3lia<Z>i!im^ia2 8?r#BS.LT^i- 
25 ^T^^S 2 2 1 {Cfc^T, T^**'*^/!'** 7fc^SrfflV^ftfc«9te 

mm<Dtmteffim%n&'rtu'£ 3 * l < , mm&z^fc-rtut 1 0 & 
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* (7) toxu («) ^i£^T&wmz-t^m\',£nz„ zpjzx<o, mm** 

5 -f^ctf?, £7ELtz1%3kmi£%'ftZ>Z£ 




10 T% ^flsjiifto^iij^^^^ LT v (i, j) j>mbfrZ 0 

m^^X^7 L yys 2 2 4 izm^X* &Mm Wife v (i, j) t^O) 

(i, j) l##HJ$*v5o 

HI v (i, j ) -m ( i , j) |/j*ka (i, j) j; 9 Vh£ »^4&£\ 
15 y^S 2 2 5tC:fc1,>T^cOjij^(7>{jjlEv (j, j) jJSotC^HSn ^ 
OiSi^fi^^5><t LT^l{«^^S'J(^$tt^> () 2> | v (i, j) -m 

(i, j) l^ka (i, j) ^7"S 2 2 6l^^t^ 

coiij^w<jgv (i, j) ^ii:$ti5 0 ZommtetafcUftt LT 

20 ^T^^S 2 2 7JC*JV^T, ±IE^f >y^S 2 2 4-S 2 2 6(DM^ 
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5 f f"ht'fe5o 121 2 9lCfc5^-t, Xf 2 2 S 3 3 3 B — S 3 3 6, 

S 2 2 7 1*. CMC, M&Wfottiik&LX&bfttzyofcmmt , 

fcot, CD-ROM2 6 0|r^$tlt^^o 
10 ±!E^ifeO?l01i8 t?tt^r-s/7'S 2 2 3 iZ&^XMMlVoft$: 1 I»Hgf2LT 1 tfc 

(D^ffciBH^^Tl^/A w^j^^t^Tli^y^S 3 3 3 A\Z.}3\,^XM 

ii, ^ifc(7^^isi^t{c^o^^T5:v^^c[R)CMM(c^^g-r5il^m^¥^mv (i, 

j) ^#iii^r<ti^^tb-r5^X-y^°S 3 3 3 B£^if„ Xf 3 3 

15 3 6{C*5V^Tfis 0 2 8lZ7fk£ivtzmm(?)ifcv (i, j) (Ditfr *) \zMM<D¥-i% 

S 3 3 3 AJCjo^T^e>n/ci^^:^^^ilif|s^*5^-CS^lC(S] C/MClfig-f 6 

sii3f5<z>5pj&i6m v (i, j) /^m^rticgm^H^o 

i^t^T^S 3 3 4l£:fc5loT\ #J^il|ft0^riii^(7j)qz^ m v (i, j) 
20 <!:^<7J>iiimi-^^SW^:iii^coPi^w¥^em (i, j) i^Iimv (i, 
j ) -m ( i , j ) I tfXmZtl* tV>%. I m v ( i , j ) -m ( i , j ) | 
k o ( i , j ) bitmZfrZo 

ilmv (i, j) -m (i, j) liSka (i, j) £<9Vh£v^§£\ * 
Tv^S 3 3 5iZ&\,^X%foMi&<D^<DmM<D¥-®mrr\v (i, j) 7>H) $ K 
25 So Cl*U;:.fc9* ^Br^Waffltf** LT«'1I^$HS 0 mU. i£lmv (i, 
j) -m (i, j) \?Aka (i, j) J:n*#^, Xf'^S 3 3 6C 
*3^T^M^Oiii^<7)ip^am v (i, j) #s 1 lC$ix5o CtHC«fci9 % ^cO 
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[%fc<Dwmi o] 

-ff-4T'i)5„ |*|3 0lC*5^T, 7>f7^ 2 2 2, S4 4 1-S 4 4 7H 

rrvt-^-^i 3oi:, ttmmft&m&LT'fte>tititatoimmk. t-mm^^ 

-ROM 2 6 OJcrd^^^TVN^o 

w-mum (i, j ) tom&mmL, ik<m& o) xm?i±ixz> w-&m 

md (K) ?r6-^Rr<!:^^EtiS*t-^^-7"^7' , s 4 4 2 ^f^S 2 2 3l: 
*3^Tmtti$n/c^M^O^i$}^ma (R) £&<d:£ (in) Kt£oT4Hatt 
R~kicWm-rz>X7 L >yy'S 4 4 3 £&fttr„ 

md(R)=^ Ki>-"M ... (9) 

m<T(R) a - (1 0) 

^T-y^S 4 4 4~S 4 4 6{C*5^Tli, IH 2 8{C^$*Wt^ | v (i, j) 
-m ( i , j) | <Dft;b <9 {il3g0>¥MMfim d (R) j&StfJVMbiv HiJIgia (i, 
j) <D^i9^^ipfHlg^¥Wmo (R) tm^^hMa ^LT, ^^-y^S 
2 2 atdfc^T^^H^flsBififett^xyrs 4 4 1 (C*3^T^&<75®ASR [Zft 
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fcc^T^^-y^S 4 4 2{C^JVnT. 1^{£uSHi&<7}#®Jfi|c R ^W^m^nm v ( i 
j) t^cO^RlC^^SITjSrilH^^ (i 
j) v (i, j) -m (i. j) | tfUVbtStl* ^W^f-J^iSttnd 

(R) /j^fefiJ^RrilC^UI^tL^o 
5 fec^T^^^S 4 4 3 (C*}^T, Xfy^S 2 2 2 JCfJI^TftfH $ *LfcSf $ 

fflf&o (i. j) WW^'ma (R) tfftmWRZk\Z%m£tlZ 0 

|c^T7sfy7 B S 4 4 4(I^PT 1 ^)-Wmd (R) /^kma (R) tit 
&£;ft5o g<W$HBmd (R) iSkma (R) J; 9 <>'J"$ vtB^, ^f^S 
4 4 5(d*DV>T^:<7;MJ!gRF^<^liimcofiirv (i, j) ^<toi:$h5o Zfi 

filmd (R) /jSkma ( R ) <fc K) V 7>f -y/S 4 4 6 f:*J^t-e« 

^Rf^60Pi^(Df^ v (i, j) *?t'<Ti :lxi:J:''j 1 -^r^uSi^c R 

. Mll^f i//S 4 4 7»>t > Pid^ f 7 7"S 4 4 4 - S 4 4 6 C)^S^ 
15 -f T cOfiglsSc R o V ^ T ?T * o jfc 7)^ £ viM'-lJ j£ £ ti 5 « . I : J?E/5 S -f T <£> f« 

o^-r5iijmo-TOfia<}:60ig(o^i*jfiSmd (r) &frm%iR r <t t-mm u -t 

20 <Z>^.cO¥J£tein d (R) rtMg^fIM^¥-#Jfittm a (R) <Z> kfflF J: «9 fc*:#l^fli«G 

**J, ±t£$*1fc<DM-m 1 0<7U9lCXf77 , S 4 4 2 t-*5V ^T^f^jii^O^^ 
^cR^cO^ilj^cDfoav (i, j) t^oo^teJcR^^i-a^^Mc^^Rl^^ 
25 S*Jt£i-Siii3tt<7)5pJ©i[m (i, j) £<&2££3JWl-*"5*"jiS*T* L^s, ^Hi{ft 
cD^-M^RF^KDiij^^^^iSm v (i, j) &*Hi U ttfcHHfcW&SHitfR F*)<7) 

®i^c^i^i£<b^^Ri;:#j£1-5^ ( \ , 

■r— Hdisv^T, ^M^o^r^Rr^i«7)=&iii^i<7)^ v (i, j) &Wofemi£(» 
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{a#RKi*tJ£i'3f!fft^ (i, j) coftjiifcR 

rtlifc'lti^Jlmm (R) SrUHiL, ^ h^ZtihcO^^^nmL, Z 

-^l— H^*J^T, Xf-7^S4 4 2T'md (R) £#J05R£(- % md (R) 
Htlmv (R) -mm (R) | ^m%^tHtX^\ 

mmomm 1 1 ] 

ht'#)5 u 1 0 T*li^^ -y 2 2 3 {C:Jul^T*t^>(fc 

£ 1 ta]Jtf2LT 1 fe<7^^ifij#<Sr^TV^^(C>Pj-L, roo^c^^Ji 1 1 T'lli: 
SQ^jSBco^Hi 9 tPHUK^s/^S 3 3 3 A ic^J^T^M^R 
^[Ej^LT^fC^^^uSiflR^^TV^o Lfc/j5oT > 7«f^S5 5l|^W 

^oT, ^"T ^7"S 5 5 5*5J;T>'S 5 5 6 {C*J^Tte, I^COM v (i, j ) CO 

ftfc>9 icitias-^TOHftm v (i, j) /r>mi<^bnz> 0 
m^miki:nx^^tz^>, tmvufevffi&miziortz A/n'^^^^^ 

mm<DBm 21 

Z ktfXZ £ iy(D^m^^Z ttfXZ 5„ 
El 3 2 li, CcO^-^^/Uf1-:p^®0OMlllS^^^ P y * l*lT'fc5 0 El 3 2 £ 

#bslt. Aflc^^/uffsfijcisEtt, si^^i 0 9, mm§mi® 2 2 0 , 

1 3, nmmm&2 3 o*3£wmmmmw2 4 o&ffi?L7> c mm&i 
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2 0 % «5W/»J»»P I 1.'3. TRfltSBigSB 2 3 0 *J £Xfi{*&tftmil1&tR 2 4 011, 

l*9^S:ltt9Ji-6^:«><0KIT*>5o 0 6Att % 0*1 3 3 tf)*-r t"s 2 izfrl-f 5 *Jg« 
^Xjy^S 1 4K*5ttS#— 7^ >^'&S£ffc^3fc#<7;>®T'&5 0 1216 Dli, 

HI 3, HI6A~|*|6E, &I3 2, *J £0*0 3 3 £r#0§ LXt&mZft ft ~j 0 X-r 
i/Blsk\-£. Xt^l 20©W^7^-^ (/°— X^t^ -/ye) 

J;t>'77^ 7 120 ^IhHEt 1 — :?Vn i o tco^SMf^^^^s^cT)- tT^-So 
/H 1 0 Kttfcfcflc&e^i-fc, 3m*?tt&*S*U W^rlBfi^^r 1 «c#<5 0 

tt, *^7i2oiaot, rn^aa?^^ oo^^n 

^MftA 1 ~Anm fc£;if3\ M1km¥i 0 0&1 0° r«t 
tclUte$ 3 6^WMA 1 - A3 6£f#5 0 J^Tcofftn^T-Ji, rtDcfc '3 
l-LT#e>ixfc3 6tfcCD^^jli^A 1 ~A 3 6 & ^ £ K a££Mt7^u 2 9 2rffr&-r 

1 3iaotHi$Mo ^r-^^s 1 OT**ofc ! Hf«jiiiflfe*j«to f ^fl5: 

anal*. si^i2tisP2 2 oiciEti$^5o *mm<njtm\ 2-cia. xt^zmiz. 
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1 2 W «t •) lc, ^ v 1 2 0 £|£|& U *ffc4fcfo: 1 0 o £ 
HI*; J^Ctiaot, ttlkm* 1 0 0 (7>tfJ ti (/^«[^7Ifij^» £ AT *r 

io i-^o 19 % #^sififeA i-A36^ i*ftmmt</mTmmm&frfo\ 

*y>V*. y h m\%L B 1 - B n &ffrfc-*-£ fc ftftQuft A 1 ~ A 3 6 //> 3 6 #CT 

6 /c * , 3, h pif^ U6ftc/i6„ ::t. -myvm. (mn & * to 5 '/a 
^f77'si4tu i*i^Lfti/vi?---^ y ^ww, 

man (mmmmm &^&^L^{i«m l</>*p»&, *t&«9f* .1 oow 

ft *j , # : ^ -fcr /uagffl £ L X it ifr &m&. ;K '/ ^ l«I £ ffl ^ T t > J; WjS, R fft 
-C^to/cJt^^ 1 0 0 <n&ftMiK£m&</)tf: y V 2 7 T'^ia-T 5 r y rf > 

2 7 x*m$i£i%tiiLteMm*, &ik&mu2 3 oicftiit^n^,, x^y^s 1 s 
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mn/mni i :i/MrK*<\ ^/^y m^mp, n^-^^f tz^^mm 

# ij 5 /K yiy Y^fifet- ov^T g£«D |% 1" 3 „ 

^Jt) &£Xfi#*?i 2 0 bmZT—y^l l O(Df5£B0li$£jRy.><5„ *i\ * 

^ 7 1 2 OcOrt^/N^^— * (a?— *^?-r^ 7 It) Koi^Tlfc^i'S, BI34 

10 "C*>-5o 1H3 4MU > 3 1 *^ 7 1 2 Ol:J:otjSit 

-5. CCO*^C 3 1 y— > 3 3td*,.t 9 <t*A£ «t ^JS^ 

■t5 9 ::<z>£ #co, 7 l 2 0 b&m-fa 3 1 £<&ffl<7Dggj$L&g-NBH-5 0 
JfcK, t>*=7\ 2 0 tleHE? 1 -— 1 0 <tOfi5:gM^(7;3hiiJ^oVNTI^-r 

20 i~5fc^(D{a-cfc6„ in 3 5Ait mte^— i o^ii^ u y zmm 

ldS^tbfc^p< 7 1 2 0 5r*-riglT**>2)o H13 5 A?r#iLT, mfcT—T" 
./HI Oc^Mtl^ (x y zJ^Jf5&) £rffl^T, *^7l 2 0 (D{\lW (x 0 , y 

0 , z o ) £r3fc#>5o $ f>(d, *^712 0(7)*U3 5<7)&\V) (Dmfefi a 

lg)3 5BJi, 5 A<Dfi 1 2 0(Dy z^ffi^0)fcft%6Zfjk-f[fiX'fo 

25 El3 5B^#f!«LT, iJ * 7 1 2 0 OTt$lll 3 5 <b , y ttH&S/jrrft 0 fcifcttS,, 

HI 3 5 C fi, HI 3 5 K<Df) ^7l20Wxy ^ffi^cOIEM££nH-[l3-e,fc<5o EH 
3 5 C£#!»LT, *^712 0^7t#3 5 t ytt^tftvt*ft5 e 
oiij, T^^^l 2 0 ^IhIIk^ — 7VH 1 0Of£g[TO£ Lt, IsHicT— ^/u 

1 1 0<7^MI1^ (x y zMfl^) £rfl?Wc#>< 7 1 2 0 czM&dfc J; tt , 0 , 
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y fc*>\ ^^Mi^^fil 1 2T?rt. ft /If a, vrt, I.IUO^ ICLTV^. 

*,\ ft/g/sfc. ©«G7 1 --^/n ioi;©>ntxmft:i;Jtti.^^7i 2 0» 

fi^^^il;4f^r^2 . 9 £flsj*1-a 0*19, *tfM*jft:£|Jt£- 

|6] (x y f-ifii^ W^^ffnj) ^e>«|KLT^<bnfc!|fe«cpii(fefc*l-t-C/j:<, j&«>± 

SWT, *ff/J? «fc < iEft^fOw 2 9 ZfttiZXZ Z> n 

0 3 6S:^iLt; Hf5Jl£«^#*1?^<£|Ri2 5 ltt, »<?;>tf^-fe/w 
3 9A»P>/jJ5 n PJffiffi«^«K*-fe/U!3;fttl2 5 1 {Z&rtZtftlr/u&sflW 

-l-ZttltXDm&WlZlto* m'3 6C0|Uffi&a*y*#*-fe/U3£|Bj2 5 1 Srqj.^«l4 0 £• 
MttH^So ^LT, rw| r Jtt2 5 4', *^M4 0l:iJE-iS:#3%&3l£#(O 
¥ffiT'^irr-rSo ti'#4om i rt>o, 4«,fr<tt4 (He WMfcw 

¥®TRtt2 5^^-rSo Sfefc, f>ik«l4 0&M£^3SfcS«&co|Hlif!Gig 

1 lc*m5&#'^-tr/U3 9lc*K£:-f5o 

1 2 -Cf# 3 6 *fcco f/Ua: ^ h Biife Bl~n36?:^l:, HJ jtj&tjg^ tf" ? 
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XfoZ>„ m 7 -CIS, 1 ow-^Hy hBSHfcB 1 fc'tt&PI^LTV^.. 1*1 3 7 *J J: 
tflfcl 7&#!!«LT, ->/U3i5/ HPSfftH I {dftifl Lt, ->^y h iSjf^ B 1 d*J--J- 

x$ z>„ nwfafc i o cm, zwis^/HmMw^iRii^^.-rrrriiLrt^wirii 

03 6fc#BlLT, tf—r-f >rMT*fi, ifi^ftfii«tS 

10 ->/l^n-.y hPHi&B l-lH 6lC^LTtf^e>o fc^7Jl 3 6 tfcco *>/Uoi * hiil 
. fft B 1 ~ B 3 6 (c3r^i-S1"'<T(0(S^^nTfia«^ fE'fe «9 S") ff^lcfc ;J? 
?-t/u3 9^*5^3 Liggett rs 6 J (C^5„ 

i.3 SO^fy/S 1 OT'tt, 1 0° r£t-*t#!^&#^L 3 6 ffciD^flqOii 
\%%:VtX, 1 2T'3 6^jCCO->^oio/ K«B 1 ~B 3 e£3=fifcLT^ 

<D*;fc>"5K: % 10° ?tiz.i±Lm-fZ>^t^teZ> 0 (K^#?t«4Seomjfe 

^ ^OjftJK^'WwfflS) C0{£i?i{;±. H13 SC^fyT'S 8(£*3ttS*-r 

^ hSSififcB 1-B3 6 <t^ix(C^i-5<5^#r£^<7)m^ vtoim^ 

rt*»9fta*ifc3£$n3o 2 0 £ [elites— 1 0 ^cofigfcWJ- 

iot, i^/l/cc y h M B 1 — B 3 6 (C*f/£i-<5 f5ZE# £ figJtoTCft {1l * =7 V) 

1*13 8li. ?}?— ^--r ^^fi^m$r^i-(2|-C**>5 0 0 3 8 £r#RBLT» feco 
25 »^ffP»l±fi:SI«* 5 #<, feC0m^^>im»^>^V\ &*5, |K|3 8<Oz.«l 
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*s*f«s^«c i o o (Dtr.femik-c*>z> n tz t ;tfri\ L £ l r 3 2 j t -J- 5 ^ , 
W±« ct 0 (C , ^^Haico^flg 1 2 "Cli, ;K— r -r >- yfttppji J; o T . 'M&tyjto 

£ 60 #*J, m^oy^mm^. J; t> ic«cfl5^JSr^xv^S fctt, 

LT*5fci\ Xt-&Qo(*v>j&Vt(n—m&Xi-t1ii «fc O^t.^^^LT^^^ij:, ^ 

I£]3 9 A*5.fcUqg3 9 Btt % H3 3«Xfy^Sl 6 {C&li 6 jK U ^> 4:j& W 
ft«£Wl*J^*SttWi-S-fe«><Z>0-CifoSo El 4 Ott, 1213 3(DXf^S 1 6i^fc 
i-tS/-Ky ^>«Oto^|ftW-r57tfe09^|T'^^ o |g|3 9Btt, IE 3 9 A 1* » 
3 9 Bg|!{;^t5i$Rl4 3a, 43b P^fc ?K !J =f > 

■e*>5„ HI 3 9 A*S«fctJ«Bl4 0 Lt, ^fyT'SAim [2^L4H^J 

^ ^ ^«lSl<£>*S*fc£^T#«> hfttz.%mWfc 1 0 0 (7) &f*ff£ 
(13 3 8£Bg) &«#tf>-^ifii (0 3 9Aflt :)OO^ffi4 la, 4 1b, 4 
lcfcttSrdl^) XWmL, &WWrffi (Hl3 9A-eJi, 3 0(7)tirjRlfffi4 4 a , 4 
4b, 44c tzrtZmm <Dm$m (ISl 3 9 A -CH, 3 ^>0)^$\^4 3a, 43 
b, 4 3 c tzii^m^) Jfc£>5 0 wHT, @3 3©^f7^SlO-Ctt10° r 
£lZ*mtoft&i&&Ltofrm&L&nX* X*ry?S 1 2TM 0° ^tWJ//Hy 
hiil^B 1~B 3 elrMUl^o ZCVtztb, ftfe&fc l o 0 ©Sflcfl^KSr* 
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¥-\MXfoZ> n 

5 t X. If , U. Ramer, " An Iterative Procedure for the Polygonal 
Approximation of Plane Curves" , CG1P, Vol. 1, pp. 2 4 4 — 2 5 6, 1 9 

^^t5;^ia t ), m&tfj^±i$.£hZ)tfy =f>%c*®m-tz>zt *>*mx 

W\3 9 B£#!$L-C, ^f-y^S A 2*3^1^^^ -/T'S A 3 (C^itt-S^^^Ui. 
^fy/S A2 "Cli» l«4 3a, 4 3 b £#5-ft0£f61 U -^(D^ft 
15 B<7>m&4 5 a, 4 5 b CDMIfj & ?R £ 0 ^ LT, $§}$lM?il4 3 a ZgrftBiUm-t 
^Zti^X^X^t.^tzm^<Dm^4 5 a\C^\,^X. m^^V h<DlS]±^BBX 

mm~rz 0 mm 4 3 b^#^Bifi(aL-c5ie*ofe^coffi^4 5 bico^-c^ 
mm<»%m$:ft & 9 0 - r t% tt^isift 43a <o#rajf?. 45at^i 
H$i5^4 3 b ^arei-s^ra^u 5 b x-r^s A3-eii, mwm4 

20 4 a <£>«tfftft4 3 a ^*tJ&-T <5TM,£ 4 5a^ §JBf ffi 4 4 b(7)M^4 3 b 

4 5a tM4 5 b tSriei|-e««»i-5^«fei LT, ^Rff^aiZr^t»TO*3 

z t izj:^>x'&bfrtim&t<Dmx\ m£f$<n&£fc&t>m\<m&m&MMxm. 

59 

BNSDOCID: <WO 9809253A1 _l_> 



WO 98/09253 

PCT/JP97/02997 

T, SUWrffiS c n t & jLU^JKfrffiiS c n t + 1 &%;L£ 0 fed 4 l £#H8 lt. ffi 
,6 a, b, c, d, e, fit »'Sc n tWi«^^fig(y LT |«f, n 
tzi><DXhZ> 0 ffi&A, R, C, D, E, F , Gfi, yjKTrfifff S c n t + 1 

SURInSS c n t + l cofjH/XA-G^WJt;fTHtF J nrv>i 0 ^LT, ftm*&iz}3 

a~ f (04 1) t. t501^iffiS c n t + 1 c/ji»|a|itt5If! ^ A-G (0 4 

m 4 i mate d t d t ropnoKBttt. r 2 j t*>5„ 

cW^^, mM&m'h(Di><D&m#iL, *om£Mzmmxtm-tz>i><Dx-b 

ra^>K«i*s» m,& c BRi^rewtJ: os^cot'. m^-b cf8isria:«T*««gi-5„ * 
femtm&c cm<D&mtt^L\,wxi<^i?H%temix ty^^t^ c:-eit m 
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m-)&&m£M$mmz .tot, @4i0^a-f tMA-Gi l t 
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